Author Archives: admin

Commit a VMware snapshot from ESXi as it doesn’t appear in vCenter or command line tools

First of all we get the ID of our VM:

# vim-cmd vmsvc/getallvms

Check his snapshots:

# vim-cmd vmsvc/snapshot.get <vmid>
Get Snapshot:

We can see that we don’t have any snapshot but that’s not true:

# ls -l *vmdk
-rw-------    1 root     root       6554112 Oct 30 20:55 vm-000001-ctk.vmdk
-rw-------    1 root     root   16559525888 Oct 30 20:55 vm-000001-delta.vmdk
-rw-------    1 root     root           393 Apr 21  2014 vm-000001.vmdk
-rw-------    1 root     root       6554112 Apr  3  2014 vm-ctk.vmdk
-rw-------    1 root     root  214748364800 Apr  3  2014 vm-flat.vmdk
-rw-------    1 root     root           585 Apr  2  2014 vm.vmdk

In order to fix it, first of all shutdown the VM as it can’t be fixed online:

# vim-cmd vmsvc/power.shutdown <vmid>

And we commit the snapshot manually from command line to a new disk (it can take some time):

# vmkfstools -i vm-000001.vmdk new.vmdk
Destination disk format: VMFS zeroedthick
Cloning disk 'vm-000001.vmdk'...
Clone: 100% done.

If the task was successfully now we overwrite our old files and delete the delta disk (I suppose that you have backups just in case something goes south):

# mv new-ctk.vmdk vm-ctk.vmdk
# mv new-flat.vmdk vm-flat.vmdk
# mv new.vmdk vm.vmdk
# rm *000001*

And finally edit your vmx/vmdk file to reflect this changes:

# grep vmdk *vmx
scsi0:0.fileName = "vm.vmdk"
# grep vmdk vm.vmdk 
RW 419430400 VMFS "vm-flat.vmdk"
changeTrackPath="vm-ctk.vmdk"

And if you powerup your VM everything should be back to normal!

If you have more than one snapshot this might be handy:

* http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004545

Fix foreman reports in Satellite 6.0

Foreman reports in Satellite 6.0 doesn’t work out of the box, first we need to setup our SMTP settings.

Error that we face without SMTP configured:

[email protected]:/var/log/foreman# /usr/sbin/foreman-rake reports:summarize
rake aborted!
getaddrinfo: Name or service not known

Tasks: TOP => reports:summarize
(See full trace by running task with --trace)

We just need to configure our settings in /etc/foreman/email.yaml like:

[email protected]:/var/log/foreman# cat /etc/foreman/email.yaml
# Outgoing email settings
production:
  delivery_method: :smtp
  smtp_settings:
    address: smtp.example.com
    port: 25
    domain: example.com
    authentication: :none

And if we run our reports once again we’ll get our email

Upgrading from a new host through Satellite 6.0 returns 403 Forbidden error

By default installation of Satellite 6.0 doesn’t update python-rhsm, if we don’t update we’ll hit this bug during a yum update:

* https://bugzilla.redhat.com/show_bug.cgi?id=1112833

Example:

https://rhs6/pulp/repos/fite_labs/DEV/rhel-6-server-x86_64-cv/content/dist/rhel/server/6/6Server/x86_64/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-6-server-rpms. Please verify its path and try again

Easiest solution is to upgrade python-rhsm package and reboot the server (or restart katello service).

[email protected]:~# yum upgrade
Loaded plugins: product-id, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package python-rhsm.x86_64 0:1.10.12-2.el7 will be updated
---> Package python-rhsm.x86_64 0:1.12.5-1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                                  Arch                                Version                                      Repository                                  Size
============================================================================================================================================================================
Updating:
 python-rhsm                              x86_64                              1.12.5-1.el7                                 katello-local                              112 k

Transaction Summary
============================================================================================================================================================================
Upgrade  1 Package

Total download size: 112 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : python-rhsm-1.12.5-1.el7.x86_64                                                                                                                          1/2 
  Cleanup    : python-rhsm-1.10.12-2.el7.x86_64                                                                                                                         2/2 
  Verifying  : python-rhsm-1.12.5-1.el7.x86_64                                                                                                                          1/2 
  Verifying  : python-rhsm-1.10.12-2.el7.x86_64                                                                                                                         2/2 

Updated:
  python-rhsm.x86_64 0:1.12.5-1.el7                                                                                                                                         
Complete!
[email protected]:~# reboot

And try again to update from a new host and that it would work! Magic!

First steps with Hammer and Red Hat Satellite 6.0

Instead of using the web interface Red Hat Satellite 6.0 offers a neat CLI utility called hammer.

In order to use it, first of all we set our default config:

# mkdir ~/.hammer
# chmod 600 ~/.hammer
# cat  ~/.hammer/cli_config.yml
   :foreman:
       :host: 'https://xxx'
       :username: 'admin'
       :password: 'xxx'

And then we’re ready to hammer!

Create a new organization and user admin into it:

# hammer organization create --name=fite.cat.lab --label=fite.cat.lab
# hammer organization add-user --user=admin --name=fite.cat.lab

And a new location:

# hammer location create --name=Catalonia
# hammer location add-user --name=Catalonia --user=admin
# hammer location add-organization --name=Catalonia --organization=fite.cat.lab

Create a domain:

# hammer domain create --name='fite.cat.lab'

Create a subnet:

# hammer subnet create --domain-ids=1 --gateway='10.1.1.1' --mask='255.255.255.0' --name='10.1.1.0/24'  --tftp-id=1 --network='10.1.1.0' --dns-primary='10.1.1.1'

Associate domain/subnet to our organization/location through the web portal:

# hammer organization add-subnet --subnet-id=1 --name='fite.cat.lab'
# hammer organization add-domain --domain-id=1 --name='fite.cat.lab'

Upload our manifest.zip (created in RH Portal) to our org and list our products:

# hammer subscription upload --file manifest.zip  --organization=fite.cat.lab
# hammer product list --organization fite.cat.lab
   ---|---------------------------------|-------------|--------------|--------------|-----------
   ID | NAME                            | DESCRIPTION | ORGANIZATION | REPOSITORIES | SYNC STATE
   ---|---------------------------------|-------------|--------------|--------------|-----------
   8  | Oracle Java for RHEL Server     |             | fite.cat.lab | 0            | not_synced
   9  | Red Hat Enterprise Linux Server |             | fite.cat.lab | 0            | not_synced
   7  | Red Hat Beta                    |             | fite.cat.lab | 0            | not_synced
   ---|---------------------------------|-------------|--------------|--------------|-----------

List all repositories included in a previous imported product:

# hammer repository-set list --organization=fite.cat.lab --product 'Red Hat Enterprise Linux Server'
   -----|-----------|------------------------------------------------------------------------------------------
   ID   | TYPE      | NAME                                                                                     
   -----|-----------|------------------------------------------------------------------------------------------
   861  | file      | Red Hat Enterprise Linux 5 Server Beta (Source ISOs)                                     
   3535 | yum       | Red Hat Enterprise Linux 5 Server Beta - Optional Productivity Applications (Debug RPMs) 
   ...

Enable repositories that we need, in my case all RHEL7 related ones:

# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - Fastrack (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - Optional (RPMs)'  
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - Extras (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - RH Common (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - Optional Fastrack (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - Supplementary (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'RHN Tools for Red Hat Enterprise Linux 7 Server (RPMs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server (ISOs)'
# hammer repository-set enable --organization fite.cat.lab --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server (Kickstart)'
# hammer repository-set enable --organization fite.cat.lab --product 'Oracle Java for RHEL Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server - Oracle Java (RPMs)'

Then we create a new product for Puppet modules in forge:

# hammer product create --name='Forge' --organization=fite.cat.lab
# hammer repository create --name='Puppet Forge' --organization=fite.cat.lab --product='Forge' --content-type='puppet' --publish-via-http=true --url=https://forge.puppetlabs.com

And finally we create a new product for EPEL packages:

# hammer product create --name='EPEL' --organization=fite.cat.lab
# hammer repository create --name='EPEL 7 - x86_64' --organization=fite.cat.lab --product='EPEL' --content-type='yum' --publish-via-http=true --url=http://dl.fedoraproject.org/pub/epel/7/x86_64/

Then we can sync all repositories that we've enabled with this simple script:

# for i in $(hammer --csv repository list --organization=fite.cat.lab  | awk -F, {'print $1'} | grep -vi '^ID'); do hammer repository synchronize --id ${i} --organization=fite.cat.lab --async; done

Create 3 environments DEV->PRE->PROD:

# hammer lifecycle-environment create --name='DEV' --prior='Library' --organization=fite.cat.lab
# hammer lifecycle-environment create --name='PRE' --prior='DEV' --organization=fite.cat.lab
# hammer lifecycle-environment create --name='PRO' --prior='PRE' --organization=fite.cat.lab

Create a daily sync plan:

# hammer sync-plan create --interval=daily --name='Daily sync' --organization=fite.cat.lab
# hammer sync-plan list --organization=fite.cat.lab

And associate this plan to our products, it must be done by sync-plan-id, not name otherwise hammer doesn't work:

# hammer product set-sync-plan --sync-plan-id=4 --organization=fite.cat.lab --name='Oracle Java for RHEL Server'
# hammer product set-sync-plan --sync-plan-id=4 --organization=fite.cat.lab --name='Red Hat Enterprise Linux Server'
# hammer product set-sync-plan --sync-plan-id=4 --organization=fite.cat.lab --name='Forge'
# hammer product set-sync-plan --sync-plan-id=4 --organization=fite.cat.lab --name='EPEL'

Create a content view for RHEL 7 server x86_64:

# hammer content-view create --name='rhel-7-server-x86_64-cv' --organization=fite.cat.lab
# for i in $(hammer --csv repository list --organization=fite.cat.lab | awk -F, {'print $1'} | grep -vi '^ID'); do hammer content-view add-repository --name='rhel-7-server-x86_64-cv' --organization=fite.cat.lab --repository-id=${i}; done

Publish this content view to Library:

# hammer content-view publish --name="rhel-7-server-x86_64-cv" --organization=fite.cat.lab --async

Promote this content view to DEV,PRE,PRO (we need to use the IDs or hammer doesn't like it, so let's get them):

# hammer content-view list --organization=fite.cat.lab 
   ----------------|---------------------------|---------------------------|-----------|---------------------------------------
   CONTENT VIEW ID | NAME                      | LABEL                     | COMPOSITE | REPOSITORY IDS                        
   ----------------|---------------------------|---------------------------|-----------|---------------------------------------
   3               | rhel-7-server-x86_64-cv   | rhel-7-server-x86_64-cv   |           | 33, 25, 31, 27, 24, 34, 26, 23, 28, 29
  ...
# hammer lifecycle-environment list --organization=fite.cat.lab
   ---|---------|--------
   ID | NAME    | PRIOR  
   ---|---------|--------
   2  | Library |        
   5  | PRO     | PRE    
   4  | PRE     | DEV    
   3  | DEV     | Library
   ---|---------|--------

# hammer content-view version promote --organization=fite.cat.lab --lifecycle-environment=3 --id=3 --async
# hammer content-view version promote --organization=fite.cat.lab --lifecycle-environment=4 --id=3 --async
# hammer content-view version promote --organization=fite.cat.lab --lifecycle-environment=5 --id=3 --async

Create a host collection for RHEL7:

# hammer host-collection create --name='RHEL 7 x86_64' --organization=fite.cat.lab

Create an activation key for our environments:

# hammer activation-key create --name='rhel-7-server-x86_64-ak-dev' --organization=fite.cat.lab --content-view='rhel-7-server-x86_64-cv' --lifecycle-environment='DEV'
# hammer activation-key create --name='rhel-7-server-x86_64-ak-pre' --organization=fite.cat.lab --content-view='rhel-7-server-x86_64-cv' --lifecycle-environment='PRE'
# hammer activation-key create --name='rhel-7-server-x86_64-ak-pro' --organization=fite.cat.lab --content-view='rhel-7-server-x86_64-cv' --lifecycle-environment='PRO'

Associate each activation key to our host collection:

# hammer activation-key add-host-collection --name='rhel-7-server-x86_64-ak-dev' --host-collection='RHEL 7 x86_64' --organization=fite.cat.lab
# hammer activation-key add-host-collection --name='rhel-7-server-x86_64-ak-pre' --host-collection='RHEL 7 x86_64' --organization=fite.cat.lab
# hammer activation-key add-host-collection --name='rhel-7-server-x86_64-ak-pro' --host-collection='RHEL 7 x86_64' --organization=fite.cat.lab

And we add all subscriptions that we have available to our keys:

   # for i in $(hammer --csv activation-key list --organization=fite.cat.lab | awk -F, {'print $1'} | grep -vi '^ID'); do for j in $(hammer --csv subscription list --organization=fite.cat.lab  | awk -F, {'print $8'} | grep -vi '^ID'); do hammer   activation-key add-subscription --id ${i} --subscription-id ${j}; done; done

Associate a partition table to OS:

# PARTID=$(hammer --csv partition-table list | grep 'Kickstart default' | awk -F, {'print $1'})
# for i in $(hammer --csv os list | awk -F, {'print $1'} | grep -vi '^ID')
  do
    hammer partition-table add-operatingsystem --id="${PARTID}" --operatingsystem-id="${i}"  
  done

Associate kickstart PXE template to OS:

# PXEID=$(hammer --csv template list | grep 'Kickstart default PXELinux' | awk -F, {'print $1'})
# SATID=$(hammer --csv template list | grep 'Satellite Kickstart Default' | awk -F, {'print $1'})
# for i in $(hammer --csv os list | awk -F, {'print $1'} | grep -vi '^ID')
  do
    hammer template add-operatingsystem --id="${PXEID}" --operatingsystem-id="${i}"
    hammer os set-default-template --id="${i}" --config-template-id="${PXEID}"
    hammer os add-config-template --id="${i}" --config-template-id="${SATID}"
    hammer os set-default-template --id="${i}" --config-template-id="${SATID}"
   done

And we create a RHEL7 hostgroup:

# MEDID=$(hammer --csv medium list | grep 'Red_Hat_7_Server_Kickstart_x86_64_7Server' | awk -F, {'print $1'})
# ENVID=$(hammer --csv environment list | grep rhel_7_server_x86_64  | grep -i dev | grep -v epel | awk -F, {'print $1'})
# PARTID=$(hammer --csv partition-table list | grep 'Kickstart default' | awk -F, {'print $1'})
# OSID=$(hammer --csv os list | grep 'RedHat 7.0' | awk -F, {'print $1'})
# CAID=1
# PROXYID=1
# hammer hostgroup create --architecture="x86_64" --domain="${DOM}" --environment-id="${ENVID}" --medium-id="${MEDID}" --name="${HGRHEL6DEV}" --subnet="${NETNAME}" --ptable-id="${PARTID}" --operatingsystem-id="${OSID}" --puppet-ca-proxy-id="${CAID}" --puppet-proxy-id="${PROXYID}"

And finally creating a new host via PXE should work...or not 🙂

Later on I'll try to post a complete script via GitHub as copy/pasting this is a nightmare, stay tuned!

Using pulp client in Satellite 6.0 from command line

In order to use pulp client we need to install it and configure it:

# yum install -y pulp-admin-client
# mkdir ~/.pulp/
# vi ~/.pulp/admin.conf
[server]
verify_ssl = false
host = <FQDN Satellite 6 server>

Then we need our pulp admin password:

# grep -i ^default_password /etc/pulp/server.conf 
default_password: <string>

And that’s it, we’re ready to use pulp client using admin user:

# pulp-admin login -u admin
Enter password: 
Successfully logged in. Session certificate will expire at Oct  2 22:53:05 2014 GMT.

Some example listing all our tasks:

# pulp-admin  tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  publish
Resources:   test_labs-EPEL-EPEL_7_-_x86_64 (repository)
State:       Successful
Start Time:  2014-09-25T21:15:21Z
Finish Time: 2014-09-25T21:15:21Z
Task Id:     c281fcac-61f5-4679-835e-72ded469c26c
...

Solve “unable to find provision template for” in Satellite 6.0

One of the tricky parts of Satellite 6.0 is that there’s not so much information about it yet.

Last day I was facing this issue:

# tail /var/log/foreman/production.log
Processing by UnattendedController#provision as */*
  Parameters: {"token"=>"2cef5164-592e-4a2b-b476-b5b48dc519e9"}
Found pxe.fite.cat.lab
Remove puppet certificate for pxe.fite.cat.lab
Adding autosign entry for pxe.fite.cat.lab
unable to find provision template for [pxe.fite.cat.lab] running [RedHat 6.5]
  Rendered text template (0.0ms)
Completed 404 Not Found in 1440ms (Views: 0.5ms | ActiveRecord: 4.2ms)

And once I was deploying a new server via PXE I could see a 404 request in Apache:

# tail /var/log/httpd/foreman_access.log 
10.1.1.110 - - [24/Sep/2014:21:40:53 +0000] "GET /unattended/provision?token=2cef5164-592e-4a2b-b476-b5b48dc519e9 HTTP/1.1" 404 77 "-" "anaconda/13.21.215"

And from server side I could only see that PXE was fine as it was booting but dracut was just timing out due to this 404.

So after some debugging I’ve noticed that I didn’t have any provision template for my OS, but Satellite 6.0 wasn’t complaining about it, so here’s a snippet that could help you out setting PXE/Provision template to a OS:

PXEID=$(hammer --csv template list | grep 'Kickstart default PXELinux' | awk -F, {'print $1'})
SATID=$(hammer --csv template list | grep 'Satellite Kickstart Default' | awk -F, {'print $1'})
for i in $(hammer --csv os list | awk -F, {'print $1'} | grep -vi '^ID')
do
    hammer template add-operatingsystem --id="${PXEID}" --operatingsystem-id="${i}"
    hammer os set-default-template --id="${i}" --config-template-id="${PXEID}"
    hammer os add-config-template --id="${i}" --config-template-id="${SATID}"
    hammer os set-default-template --id="${i}" --config-template-id="${SATID}"
done

Enable tftp smart proxy on Red Hat Satellite Capsule Server 6.0

Adding TFTP Smart Proxy:

  # katello-installer --capsule-tftp=true
  # systemctl restart foreman
  # systemctl restart foreman-proxy

Enable UDP port for TFTP:

  # firewall-cmd --permanent --zone=public --add-service=tftp
  # systemct restart firewalld

Refreshing features from our Capsule Server:

  # for i in $(hammer --csv capsule list | awk -F, {'print $1'} | grep -v ^Id); do hammer proxy refresh-features --id=${i}; done

Now if we want to deploy a server via PXE we need to configure a few more options like domain, subnet, etc.

Let’s see it another day.

How to install Red Hat Satellite 6.0 with RHEL7 in disconnected mode

In order to install RHS6, first of all we mount our satellite iso:

  # mkdir /root/rhs6
  # mount -o loop satellite-6.0.4-rhel-7-x86_64-dvd.iso /root/rhs6

And as we’re in a disconnected mode, we don’t have access to Red Hat Software Collections and RHS6 depends on ruby193. As the installer doesn’t do it automatically, we create a local repo for it.
It would be nice to report it to RH but I’m lazy 🙂

   # cat /etc/yum.repos.d/rhel7rhscl.repo
   [rhel7rhscl]
   name = RHEL7 RHSCL
   baseurl = file:///root/rhs6/RHCSL/
   enabled = 1

And then we can proceed to install RHS6 and configure it:

  # cd /root/rhs6
  # ./install_packages
  # katello-installer
  * Katello is running at https://xxxx
      Initial credentials are admin / xxxxx
  * Capsule is running at https://xxxx:9090
  * To install additional capsule on separate machine continue by running:"
      capsule-certs-generate --capsule-fqdn "$CAPSULE" --certs-tar "~/$CAPSULE-certs.tar"
  The full log is at /var/log/katello-installer/katello-installer.log

And then we need to open some ports in order to make everything work:

  # firewall-cmd --permanent --zone=public --add-service=http
  # firewall-cmd --permanent --zone=public --add-service=https
  # firewall-cmd --permanent --zone=public --add-port=5671/tcp
  # firewall-cmd --permanent --zone=public --add-port=8080/tcp
  # firewall-cmd --permanent --zone=public --add-port=8140/tcp
  # firewall-cmd --permanent --zone=public --add-port=9090/tcp
  # firewall-cmd --permanent --zone=public --add-port=9200/tcp
  # systemctl reload firewalld

As a first step you can login to RHS through its web interface using admin as user and the autogenerated password that katello-installer provided us:

* https://xxxx

RHS6

DNF (Experimental Yum Replacement)

In order to test DNF (Dandified Yum) we can follow these steps:

# yum install -y dnf
# dnf install fedora-release-rawhide
# dnf upgrade --enablerepo=rawhide
# cat /etc/fedora-release 
Fedora release 22 (Rawhide)

Take into account that this also will upgrade our machine to Fedora Rawhide.

As a command reference it’s almost equal to yum syntaxis:

List of Main Commands

autoerase                 
check-update              Check for available package upgrades
clean                     Remove cached data
distro-sync               Synchronize installed packages to the latest available versions
downgrade                 downgrade a package
erase                     Remove a package or packages from your system
group                     Display, or use, the groups information
help                      Display a helpful usage message
history                   Display, or use, the transaction history
info                      Display details about a package or group of packages
install                   Install a package or packages on your system
list                      List a package or groups of packages
makecache                 Generate the metadata cache
provides                  Find what package provides the given value
reinstall                 reinstall a package
repolist                  Display the configured software repositories
repository-packages       Run commands on top of all packages in given repository
search                    Search package details for the given string
updateinfo                Display advisories about packages
upgrade                   Upgrade a package or packages on your system
upgrade-to                Upgrade a package on your system to the specified version

Eudyptula Challenge completed

No updates means that I was busy 🙂

So finally I got to finish Eudyptula challenge and I would like to recommend you once again to try it and finish it.

I’ve learned so many things about the kernel that without the “guidance” of little I wouldn’t be aware of.

And if you’re curious that’s a quote from little:

Very nice job, you are now done!

If you were curious, you are the 50th person to complete the series of tasks, with over 7100 people currently attempting it. It’s not a race, but you’ve done a great job.