Friday, 20 April 2018

Using/Verifying the Autoscale service from Apps Manager UI in 5 minutes

Recently at a customer site I was asked to show how the Autoscale service shipped by default with Pivotal Cloud Foundry would work. Here is how we demoed that in less then 5 minutes.

1. Select an application to Autoscale and click on the "Autoscaling" radio option.


2. Select "Manage Autoscaling" link as shown below.


3. Set the maximum instance limit to "4" and click Save as shown below. You can also set minimum to 1 instance if you want to which will make it easier to verify the scaling of instances as one instance can easily be put under pressure.


4. Now lets set a "Scaling Rule" by clicking on the "Edit" link as shown below.


5. Now lets add a CPU rule by clicking on the "Add" link as shown below.


6. Now define a CPU rule as shown below and click on Save. Don't forget to make it active using the radio option. In this example we use very low threshold BUT it would be better to increase this to something more realistic like 30% and 60% respectively.




Now at this point we are ready to test the Autoscale service BUT to do that we are going to have to create some load. Many different ways to do that but "ab" on my Mac was the fastest way.

8. Create some load on an endpoint for your application to force CPU utilization to increase as shown below

pasapicella@pas-macbook:~$ ab -n 10000 -c 25 http://springboot-actuator-appsmanager-delightful-jaguar.cfapps.io/employees
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking springboot-actuator-appsmanager-delightful-jaguar.cfapps.io (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

....

9. If you return to Apps Manager UI soon enough you will see that the Autoscale service has fired events to add more instances as per the screen shots below.




It's worth noting that the CF CLI Plugin for Autoscale can also show us what we have defined as as shown below. More information on this plugin is as follows

https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler-cli.html#install

View which applications are using the Autoscaler service:

pasapicella@pas-macbook:~$ cf autoscaling-apps
Presenting autoscaler apps in org apples-pivotal-org / space development as papicella@pivotal.io
OK
Name                              Guid                                   Enabled   Min Instances   Max Instances
springboot-actuator-appsmanager   6c137fea-6a99-4069-8031-a2aa3978804c   true      2               4

View events for an application that has Autoscaler service bound to it:

pasapicella@pas-macbook:~$ cf autoscaling-events springboot-actuator-appsmanager
Presenting autoscaler events for app springboot-actuator-appsmanager for org apples-pivotal-org / space development as papicella@pivotal.io
OK
Time                   Description
2018-04-20T09:56:30Z   Scaled down from 3 to 2 instances. All metrics are currently below minimum thresholds.
2018-04-20T09:55:56Z   Scaled down from 4 to 3 instances. All metrics are currently below minimum thresholds.
2018-04-20T09:54:46Z   Can not scale up. At max limit of 4 instances. Current CPU of 20.75% is above upper threshold of 8.00%.
2018-04-20T09:54:11Z   Can not scale up. At max limit of 4 instances. Current CPU of 30.53% is above upper threshold of 8.00%.
2018-04-20T09:53:36Z   Can not scale up. At max limit of 4 instances. Current CPU of 32.14% is above upper threshold of 8.00%.
2018-04-20T09:53:02Z   Can not scale up. At max limit of 4 instances. Current CPU of 31.51% is above upper threshold of 8.00%.
2018-04-20T09:52:27Z   Scaled up from 3 to 4 instances. Current CPU of 19.59% is above upper threshold of 8.00%.
2018-04-20T09:51:51Z   Scaled up from 2 to 3 instances. Current CPU of 8.99% is above upper threshold of 8.00%.
2018-04-20T09:13:24Z   Scaling from 1 to 2 instances: app below minimum instance limit
2018-04-20T09:13:23Z   Enabled autoscaling.

More Information

https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler-cli.html#install

https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler.html

Tuesday, 10 April 2018

Spring Cloud Services CF CLI Plugin

The Spring Cloud Services plugin for the Cloud Foundry Command Line Interface tool (cf CLI) adds commands for interacting with Spring Cloud Services service instances. It provides easy access to functionality relating to the Config Server and Service Registry; for example, it can be used to send values to a Config Server service instance for encryption or to list all applications registered with a Service Registry service instance.

Here is a simple example of how we can view various bound apps for a Service Registry

1. Install the CF CLI Plugin for Spring Cloud Services using the link below

$ cf add-plugin-repo CF-Community https://plugins.cloudfoundry.org


$ cf install-plugin -r CF-Community "Spring Cloud Services"

2. Now in Apps Manager UI we have a Service Registry instance with some bound micro services as shown below



3. Now we can use the SCS CF CLI Plugin to also get this information

pasapicella@pas-macbook:~$ cf service-registry-list eureka-service
Listing service registry eureka-service in org apples-pivotal-org / space scs-demo as papicella@pivotal.io...
OK

Service instance: eureka-service
Server URL: https://eureka-fcf42b1c-6b85-444c-9a43-fee82f2c68c3.cfapps.io/

eureka app name cf app name    cf instance index zone      status
EDGE-SERVICE    edge-service   0                 cfapps.io UP
COFFEE-SERVICE  coffee-service 0                 cfapps.io UP

The full list of plugin commands are as shown in the screen shot below. 

Note: Use "cf plugins" to get this list once installed


More Information

http://docs.pivotal.io/spring-cloud-services/1-5/common/cf-cli-plugin.html

Wednesday, 4 April 2018

Deploying my first Pivotal Container Service (PKS) workload to my PKS cluster

If you followed along on the previous blogs you would of installed PKS 1.0 on GCP (Google Cloud Platform) and created your first PKS cluster and wired it into kubectl as well as provided an external load balancer as per the previous two posts.

Previous posts:

Install Pivotal Container Service (PKS) on GCP and getting started
http://theblasfrompas.blogspot.com.au/2018/04/install-pivotal-container-service-pks.html

Wiring kubectl / Setup external LB on GCP into Pivotal Container Service (PKS) clusters to get started
http://theblasfrompas.blogspot.com.au/2018/04/wiring-kubectl-setup-external-lb-on-gcp.html

So lets now create our first workload as shown below

1. Download YML demo from here

https://github.com/cloudfoundry-incubator/kubo-ci/blob/master/specs/nginx-lb.yml

2. Deploy as shown below

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS/demo-workload$ kubectl create -f nginx-lb.yml
service "nginx" created
deployment "nginx" created

3. Check current status

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS/demo-workload$ kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
nginx-679dc9c764-8cwzq   1/1       Running   0          22s
nginx-679dc9c764-p8tf2   1/1       Running   0          22s
nginx-679dc9c764-s79mp   1/1       Running   0          22s

4. Wait for External IP address of the nginx service to be assigned

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS/demo-workload$ kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.100.200.1               443/TCP        17h
nginx        LoadBalancer   10.100.200.143   35.189.23.119   80:30481/TCP   1m

5. In a browser access the K8's workload as follows, using the external IP

http://35.189.23.119



More Info

https://docs.pivotal.io/runtimes/pks/1-0/index.html

Wiring kubectl / Setup external LB on GCP into Pivotal Container Service (PKS) clusters to get started

Now that I have PCF 2.1 running with PKS 1.0 installed and a cluster up and running how would I get started accessing that cluster? Here are the steps for GCP (Google Cloud Platform) install of PCF 2.1 with PKS 1.0. It goes through the requirements around an External LB for the cluster as well as wiring kubectl into the cluster to get started creating deployments.

Previous blog as follows:

http://theblasfrompas.blogspot.com.au/2018/04/install-pivotal-container-service-pks.html

1. First we will want an external Load Balancer for our K8's clusters which will need to exist and it would be a TCP Load balancer using Port 8443 which is the port the master node would run on. The external IP address is what you will need to use in the next step



2. Create a Firewall Rule for the LB with details as follows.

Note: the LB name is "pks-cluster-api-1". Make sure to include the network tag and select the network you installed PKS on.

  • Network: Make sure to select the right network. Choose the value that matches with the VPC Network name you installed PKS on
  • Ingress - Allow
  • Target: pks-cluster-api-1
  • Source: 0.0.0.0/0
  • Ports: tcp:8443





3. Now you could easily just create a cluster using the external IP address from above or use a DNS entry which is mapped to the external IP address which is what I have done so I have use a FQDN instead

pasapicella@pas-macbook:~$ pks create-cluster my-cluster --external-hostname cluster1.pks.pas-apples.online --plan small

Name:                     my-cluster
Plan Name:                small
UUID:                     64a086ce-c94f-4c51-95f8-5a5edb3d1476
Last Action:              CREATE
Last Action State:        in progress
Last Action Description:  Creating cluster
Kubernetes Master Host:   cluster1.pks.pas-apples.online
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  In Progress


4. Now just wait a while while it creates a VM's and runs some tests , it's roughly around 10 minutes. Once done you will see the cluster as created as follows

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ pks list-clusters

Name        Plan Name  UUID                                  Status     Action
my-cluster  small      64a086ce-c94f-4c51-95f8-5a5edb3d1476  succeeded  CREATE

5. Now one of the VM's created would be the master Vm for the cluster , their a few ways to determine the master VM as shown below.

5.1. Use GCP Console VM instances page and filter by "master"



5.2. Run a bosh command to view the VM's of your deployments. We are interested in the VM's for our cluster service. The master instance is named as "master/ID" as shown below.

$ bosh -e gcp vms --column=Instance --column "Process State" --column "VM CID"

Task 187. Done

Deployment 'service-instance_64a086ce-c94f-4c51-95f8-5a5edb3d1476'

Instance                                     Process State  VM CID
master/13b42afb-bd7c-4141-95e4-68e8579b015e  running        vm-4cfe9d2e-b26c-495c-4a62-77753ce792ca
worker/490a184e-575b-43ab-b8d0-169de6d708ad  running        vm-70cd3928-317c-400f-45ab-caf6fa8bd3a4
worker/79a51a29-2cef-47f1-a6e1-25580fcc58e5  running        vm-e3aa47d8-bb64-4feb-4823-067d7a4d4f2c
worker/f1f093e2-88bd-48ae-8ffe-b06944ea0a9b  running        vm-e14dde3f-b6fa-4dca-7f82-561da9c03d33

4 vms

6. Attach the VM to the load balancer backend configuration as shown below.



7. Now we can get the credentials from PKS CLI and pass them to kubectl as shown below

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ pks get-credentials my-cluster

Fetching credentials for cluster my-cluster.
Context set for cluster my-cluster.

You can now switch between clusters by using:
$kubectl config use-context

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl cluster-info
Kubernetes master is running at https://cluster1.pks.domain-name:8443
Heapster is running at https://cluster1.pks.domain-name:8443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://cluster1.pks.domain-name:8443/api/v1/namespaces/kube-system/services/kube-dns/proxy
monitoring-influxdb is running at https://cluster1.pks.domain-name:8443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy

8. To verify it worked for you here are some commands you would run. The "kubectl cluster-info" is one of those.

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl get componentstatus
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl get pods
No resources found.

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl get deployments
No resources found.

9. Finally lets start the Kubernetes UI to monitor this cluster. We do that as easily as this.

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl proxy
Starting to serve on 127.0.0.1:8001  

The UI URL requires you to append /ui to the url above

Eg: http://127.0.0.1:8001/ui

Note: It will prompt you for the kubectl config file which would be in the $HOME/.kube/config file. Failure to present this means the UI won't show you much and give lost of warnings




More Info

https://docs.pivotal.io/runtimes/pks/1-0/index.html

Install Pivotal Container Service (PKS) on GCP and getting started

With the release of Pivotal Cloud Foundry 2.1 (PCF) I decided this time to install Pivotal Application Service (PAS) as well as Pivotal Container Service (PKS) using the one Bosh Director which isn't recommended for production installs BUT ok for dev installs. Once installed you will have both the PAS tile and PKS tile as shown below.

https://content.pivotal.io/blog/pivotal-cloud-foundry-2-1-adds-cloud-native-net-envoy-native-service-discovery-to-boost-your-transformation


So here is how to get started with PKS once it's installed

1. Create a user for the PKS client to login with.

1.1. ssh into the ops manager VM

1.2. Target the UAA endpoint for PKS this was setup in the PKS tile

ubuntu@opsman-pcf:~$ uaac target https://PKS-ENDPOINT:8443 --skip-ssl-validation
Unknown key: Max-Age = 86400

Target: https://PKS-ENDPOINT:8443

1.3. Authenticate with UAA using the secret you retrieve from the PKS tile / Credentials tab as shown in the image below. Run the following command, replacing UAA-ADMIN-SECRET with your UAA admin secret

ubuntu@opsman-pcf:~$ uaac token client get admin -s UAA-ADMIN-SECRET
Unknown key: Max-Age = 86400

Successfully fetched token via client credentials grant.
Target: https://PKS-ENDPIONT:8443
Context: admin, from client admin



1.4. Create an ADMIN user as shown below using the UAA-ADMIN-SECRET password obtained form ops manager UI as shown above

ubuntu@opsman-pcf:~$ uaac user add pas --emails papicella@pivotal.io -p PASSWD
user account successfully added

ubuntu@opsman-pcf:~$ uaac member add pks.clusters.admin pas
success

2. Now lets login using the PKS CLI with a new admin user we created

pasapicella@pas-macbook:~$ pks login -a PKS-ENDPOINT -u pas -p PASSWD -k

API Endpoint: pks-api.pks.pas-apples.online
User: pas

3. You can test whether you have a DNS issue with a command as follows. 

Note: A test as follows determines any DNS issues you may have

pasapicella@pas-macbook:~$ nc -vz PKS-ENDPOINT 8443
found 0 associations
found 1 connections:
     1: flags=82
outif en0
src 192.168.1.111 port 62124
dst 35.189.1.209 port 8443
rank info not available
TCP aux info available

Connection to PKS-ENDPOINT port 8443 [tcp/pcsync-https] succeeded!

4. You can run a simple command to verify your connected as follows, below shows no K8's clusters exist at this stage

pasapicella@pas-macbook:~$ pks list-clusters

Name  Plan Name  UUID  Status  Action

You can use PKS CLI to create a new cluster, view clusters, resize clusters etc

pasapicella@pas-macbook:~$ pks

The Pivotal Container Service (PKS) CLI is used to create, manage, and delete Kubernetes clusters. To deploy workloads to a Kubernetes cluster created using the PKS CLI, use the Kubernetes CLI, kubectl.

Version: 1.0.0-build.3

Note: The PKS CLI is under development, and is subject to change at any time.

Usage:
  pks [command]

Available Commands:
  cluster         View the details of the cluster
  clusters        Show all clusters created with PKS
  create-cluster  Creates a kubernetes cluster, requires cluster name and an external host name
  delete-cluster  Deletes a kubernetes cluster, requires cluster name
  get-credentials Allows you to connect to a cluster and use kubectl
  help            Help about any command
  login           Login to PKS
  logout          Logs user out of the PKS API
  plans           View the preconfigured plans available
  resize          Increases the number of worker nodes for a cluster

Flags:
  -h, --help      help for pks
      --version   version for pks

Use "pks [command] --help" for more information about a command.

5. You would create a cluster as follows now you have logged in and yu will get aK8's cluster to begin working with

pasapicella@pas-macbook:~$ pks create-cluster my-cluster --external-hostname EXT-LB-HOST --plan small

Name:                     my-cluster
Plan Name:                small
UUID:                     64a086ce-c94f-4c51-95f8-5a5edb3d1476
Last Action:              CREATE
Last Action State:        in progress
Last Action Description:  Creating cluster
Kubernetes Master Host:   cluster1.FQDN
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  In Progress

Finally when done you will see "Last Action:" as "succeeded" as shown below

pasapicella@pas-macbook:~$ pks cluster my-cluster

Name:                     my-cluster
Plan Name:                small
UUID:                     64a086ce-c94f-4c51-95f8-5a5edb3d1476
Last Action:              CREATE
Last Action State:        succeeded
Last Action Description:  Instance provisioning completed
Kubernetes Master Host:   cluster1.FQDN
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  MASTER-IP-ADDRESS

More Info

https://docs.pivotal.io/runtimes/pks/1-0/index.html


Thursday, 15 March 2018

Pivotal Cloud Foundry Healthwatch for Pivotal Cloud Foundry 2.0 on GCP

I decided to eventually install PCF Healthwatch on my Google Cloud Platform PCF 2.0 instance. Installing it is straight forward using Ops Manager UI and once installed it will look like this.

Note: This is PCF 2.0 on GCP


Once installed the application for the Web UI end point would be as follows. The login username and password is the UAA admin user . By default the property "healthwatch.read" credential is given to this user only. You can always create a new user that has this credential role if you like.

https://healthwatch.SYSTEM-DOMAIN

The main page has various useful information and more then enough to show you what's happening in your PCF instance as shown below.



Clicking on any of the headings for each tile you can get more detailed information. The two screen shots below show some CF CLI command history tests like a "cf push" , "cf logs" and also what is happening within the Diego cells in terms of "Memory, Disk and the Containers" themselves.




More Information

https://docs.pivotal.io/pcf-healthwatch/1-1/index.html

Sunday, 11 March 2018

Just gave CFDEV a quick test and it's easy and includes BOSH!!!!

CF Dev is a new distribution of Cloud Foundry designed to run on a developer’s laptop or workstation using native hypervisors and a fully functional BOSH Director.

I decided to give it a test run today and it's fast and easy full CF experience deployed through the CF CLI plugin as described in the GitHub project

  https://github.com/pivotal-cf/cfdev

Here we run some bosh commands once it's up and running. You can't run BOSH commands without first setting your ENV to use the correct bosh director which you do as follows

  $ eval "$(cf dev bosh env)"

pasapicella@pas-macbook:~/apps/ENV/cfdev$ bosh deployments
Using environment '10.245.0.2' as client 'admin'

Name  Release(s)                    Stemcell(s)                                          Team(s)  Cloud Config
cf    binary-buildpack/1.0.15       bosh-warden-boshlite-ubuntu-trusty-go_agent/3468.17  -        latest
      bosh-dns/0.2.0
      capi/1.46.0
      cf-mysql/36.10.0
      cf-networking/1.9.0
      cf-smoke-tests/40
      cf-syslog-drain/5
      cflinuxfs2/1.179.0
      consul/191
      diego/1.32.1
      dotnet-core-buildpack/1.0.32
      garden-runc/1.10.0
      go-buildpack/1.8.15
      grootfs/0.30.0
      java-buildpack/4.7.1
      loggregator/99
      nats/22
      nodejs-buildpack/1.6.13
      php-buildpack/4.3.46
      python-buildpack/1.6.4
      routing/0.169.0
      ruby-buildpack/1.7.8
      staticfile-buildpack/1.4.20
      statsd-injector/1.0.30
      uaa/53.3

1 deployments

Succeeded

pasapicella@pas-macbook:~/apps/ENV/cfdev$ bosh stemcells
Using environment '10.245.0.2' as client 'admin'

Name                                         Version   OS             CPI  CID
bosh-warden-boshlite-ubuntu-trusty-go_agent  3468.17*  ubuntu-trusty  -    54a8d4c1-5a02-4d89-5648-1132914a0cb8

(*) Currently deployed

1 stemcells

Succeeded

You can simply use the CF CLI as follows once you target the correct API endpoint and login as follows

pasapicella@pas-macbook:~/apps/ENV/cfdev$ cf api https://api.v3.pcfdev.io --skip-ssl-validation
Setting api endpoint to https://api.v3.pcfdev.io...
OK

api endpoint:   https://api.v3.pcfdev.io
api version:    2.100.0
Not logged in. Use 'cf login' to log in.

and to log in ...

pasapicella@pas-macbook:~/apps/ENV/cfdev$ cf login -o cfdev-org -u admin -p admin
API endpoint: https://api.v3.pcfdev.io
Authenticating...
OK

Targeted org cfdev-org

Targeted space cfdev-space

API endpoint:   https://api.v3.pcfdev.io (API version: 2.100.0)
User:           admin
Org:            cfdev-org
Space:          cfdev-space

Sunday, 4 March 2018

Spring boot 2 Actuator Support and Pivotal Cloud Foundry 2.0

With Spring Boot Actuator you get production-ready features to your application. The main benefit of this library is that we can get production grade tools without having to actually implement these features ourselves.


Actuator is mainly used to expose operational information about the running application – health, metrics, info, dump, env, etc. It uses HTTP endpoints or JMX beans to enable us to interact with it.

In this post we will show how Spring Boot 2.0 Actuator endpoints are automatically integrated into Pivotal Cloud Foundry Apps Manager.

1. Clone the following project as shown below

pasapicella@pas-macbook:~/temp$ git clone https://github.com/papicella/springboot-actuator-2-demo.git
Cloning into 'springboot-actuator-2-demo'...
remote: Counting objects: 57, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 57 (delta 0), reused 6 (delta 0), pack-reused 48
Unpacking objects: 100% (57/57), done.

2. Package as follows

pasapicella@pas-macbook:~/temp/springboot-actuator-2-demo$ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building springboot-autuator-2-demo 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ springboot-autuator-2-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 1 resource

...

[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ springboot-autuator-2-demo ---
[INFO] Building jar: /Users/pasapicella/temp/springboot-actuator-2-demo/target/springboot-autuator-2-demo-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.0.0.M7:repackage (default) @ springboot-autuator-2-demo ---
[INFO]
[INFO] --- maven-dependency-plugin:3.0.1:unpack (unpack) @ springboot-autuator-2-demo ---
[INFO] Configured Artifact: com.example:springboot-autuator-2-demo:0.0.1-SNAPSHOT:jar
[INFO] Unpacking /Users/pasapicella/temp/springboot-actuator-2-demo/target/springboot-autuator-2-demo-0.0.1-SNAPSHOT.jar to /Users/pasapicella/temp/springboot-actuator-2-demo/target/dependency with includes "" and excludes ""
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.650 s
[INFO] Finished at: 2018-03-04T21:04:08+11:00
[INFO] Final Memory: 46M/594M
[INFO] ------------------------------------------------------------------------

3. Deploy as follows

pasapicella@pas-macbook:~/temp/springboot-actuator-2-demo$ cf push
Pushing from manifest to org apples-pivotal-org / space development as papicella@pivotal.io...
Using manifest file /Users/pasapicella/temp/springboot-actuator-2-demo/manifest.yml
Getting app info...
Updating app with these attributes...
  name:                springboot-actuator-appsmanager
  path:                /Users/pasapicella/temp/springboot-actuator-2-demo/target/springboot-autuator-2-demo-0.0.1-SNAPSHOT.jar
  buildpack:           client-certificate-mapper=1.5.0_RELEASE container-security-provider=1.13.0_RELEASE java-buildpack=v4.9-offline-https://github.com/cloudfoundry/java-buildpack.git#830f4c3 java-main java-opts java-security jvmkill-agent=1.12.0_RELEASE open-jdk-l...
  command:             JAVA_OPTS="-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.12.0_RELEASE=printHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR -Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/open_jdk_jre/lib/ext -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS" && CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-3.10.0_RELEASE -totMemory=$MEMORY_LIMIT -stackThreads=250 -loadedClasses=17785 -poolType=metaspace -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp $PWD/. org.springframework.boot.loader.JarLauncher
  disk quota:          1G
  health check type:   port
  instances:           1
  memory:              1G
  stack:               cflinuxfs2
  routes:
    springboot-actuator-appsmanager-forgiving-camel.cfapps.io

Updating app springboot-actuator-appsmanager...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...

...

Waiting for app to start...

name:              springboot-actuator-appsmanager
requested state:   started
instances:         1/1
usage:             1G x 1 instances
routes:            springboot-actuator-appsmanager-forgiving-camel.cfapps.io
last uploaded:     Sun 04 Mar 21:07:03 AEDT 2018
stack:             cflinuxfs2
buildpack:         client-certificate-mapper=1.5.0_RELEASE container-security-provider=1.13.0_RELEASE java-buildpack=v4.9-offline-https://github.com/cloudfoundry/java-buildpack.git#830f4c3
                   java-main java-opts java-security jvmkill-agent=1.12.0_RELEASE open-jdk-l...
start command:     JAVA_OPTS="-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.12.0_RELEASE=printHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR
                   -Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/open_jdk_jre/lib/ext -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security
                   $JAVA_OPTS" && CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-3.10.0_RELEASE -totMemory=$MEMORY_LIMIT -stackThreads=250 -loadedClasses=17785
                   -poolType=metaspace -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT
                   eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp $PWD/. org.springframework.boot.loader.JarLauncher

     state     since                  cpu      memory         disk           details
#0   running   2018-03-04T10:08:16Z   196.2%   385.7M of 1G   157.4M of 1G

4. The application.yml exposes all methods and is totally unsecure so you would not want to do this in a production application. The application is deployed using an application.yml as follows.

spring:
  application:
    name: PCFSpringBootActuatorDemo
  jpa:
    hibernate:
      ddl-auto: update
management:
  endpoint:
    health:
      show-details: true
  endpoints:
    web:
      expose: '*'
      enabled: true
    jmx:
      expose: '*'
      enabled: true

Once deployed Pivotal Cloud Foundry Apps Manager will show the Spring Icon and use the Actuator endpoints.





Lets invoke some of the Actuator endpoints using HTTPIE or CURL if you like. Remember we have exposed all web endpoints allowing us to do this. One thing that has changed form Actuator 1.x to 2.0 is the endpoints are now mapped to /actuator out of the box. You can get all that are available endpoints just by invoking /actuator as shown below using a GET RESTful call.

pasapicella@pas-macbook:~/temp/springboot-actuator-2-demo$ http http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/vnd.spring-boot.actuator.v2+json;charset=UTF-8
Date: Sun, 04 Mar 2018 10:13:38 GMT
X-Vcap-Request-Id: 22116a58-f689-4bd9-448c-023bae2ed5ec
transfer-encoding: chunked

{
    "_links": {
        "auditevents": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/auditevents",
            "templated": false
        },
        "beans": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/beans",
            "templated": false
        },
        "conditions": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/conditions",
            "templated": false
        },
        "configprops": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/configprops",
            "templated": false
        },
        "env": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/env",
            "templated": false
        },
        "env-toMatch": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/env/{toMatch}",
            "templated": true
        },
        "health": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/health",
            "templated": false
        },
        "heapdump": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/heapdump",
            "templated": false
        },
        "info": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/info",
            "templated": false
        },
        "loggers": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/loggers",
            "templated": false
        },
        "loggers-name": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/loggers/{name}",
            "templated": true
        },
        "mappings": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/mappings",
            "templated": false
        },
        "metrics": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/metrics",
            "templated": false
        },
        "metrics-requiredMetricName": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/metrics/{requiredMetricName}",
            "templated": true
        },
        "scheduledtasks": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/scheduledtasks",
            "templated": false
        },
        "self": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator",
            "templated": false
        },
        "threaddump": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/threaddump",
            "templated": false
        },
        "trace": {
            "href": "http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/trace",
            "templated": false
        }
    }
}

pasapicella@pas-macbook:~/temp/springboot-actuator-2-demo$ http http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/health
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 183
Content-Type: application/vnd.spring-boot.actuator.v2+json;charset=UTF-8
Date: Sun, 04 Mar 2018 10:16:03 GMT
X-Vcap-Request-Id: be45c751-b77d-4e7c-77b6-0d7affa0fe16

{
    "details": {
        "db": {
            "details": {
                "database": "H2",
                "hello": 1
            },
            "status": "UP"
        },
        "diskSpace": {
            "details": {
                "free": 908681216,
                "threshold": 10485760,
                "total": 1073741824
            },
            "status": "UP"
        }
    },
    "status": "UP"
}

pasapicella@pas-macbook:~/temp/springboot-actuator-2-demo$ http http://springboot-actuator-appsmanager-forgiving-camel.cfapps.io/actuator/trace
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/vnd.spring-boot.actuator.v2+json;charset=UTF-8
Date: Sun, 04 Mar 2018 10:19:46 GMT
X-Vcap-Request-Id: e0c86f51-dcc4-4349-41d2-6b603677c3f4
transfer-encoding: chunked

{
    "traces": [
        {
            "info": {
                "headers": {
                    "request": {
                        "accept": "*/*",
                        "accept-encoding": "gzip, deflate",
                        "host": "springboot-actuator-appsmanager-forgiving-camel.cfapps.io",
                        "user-agent": "HTTPie/0.9.9",
                        "x-b3-spanid": "0ad427a9f13bad0c",
                        "x-b3-traceid": "0ad427a9f13bad0c",
                        "x-cf-applicationid": "c1e50a41-5e1e-475f-b9e6-116a7acd98a2",
                        "x-cf-instanceid": "db74a5d2-ac72-4c45-539a-118f",
                        "x-cf-instanceindex": "0",
                        "x-forwarded-port": "80",
                        "x-forwarded-proto": "http",
                        "x-request-start": "1520158694338",
                        "x-vcap-request-id": "5f1e5572-a841-4e3f-4b6f-2cfd0c0ccc8e"
                    },
                    "response": {
                        "Content-Type": "application/vnd.spring-boot.actuator.v2+json;charset=UTF-8",
                        "Date": "Sun, 04 Mar 2018 10:18:14 GMT",

...

More Information

https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html

Thursday, 18 January 2018

Pivotal Cloud Foundry App Instance Routing in HTTP Headers

Developers who want to obtain debug data for a specific instance of an app can use the HTTP header X-CF-APP-INSTANCE to make a request to an app instance. To demonstrate how we can write a Spring Boot application which simply outputs the current CF app index so we are sure we are hitting the right application container.

Simplest way to do that is to define a RestController using Spring Boot as follows which then enables us to get the current application index and verify we are hitting the right container instance.

  
package com.example.pas;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class DemoRest
{
    private final String ip;
    private final String index;

    @Autowired
    public DemoRest
            (@Value("${CF_INSTANCE_IP:127.0.0.1}") String ip, 
             @Value("${CF_INSTANCE_INDEX:0}") String index) {
        this.ip = ip;
        this.index = index;
    }

    @RequestMapping("/")
    public InstanceDetail getAppDetails()
    {
        InstanceDetail instanceDetail = new InstanceDetail(ip, index);

        return instanceDetail;
    }
}

So with the application deployed as see we have 3 instances as follows

pasapicella@pas-macbook:~$ cf app pas-pcf-routingdemo
Showing health and status for app pas-pcf-routingdemo in org pivot-papicella / space dotnet as papicella@pivotal.io...

name:                pas-pcf-routingdemo
requested state:     started
instances:           3/3
isolation segment:   main
usage:               756M x 3 instances
routes:              pas-pcf-routingdemo-incidental-curia.pcfbeta.io
last uploaded:       Thu 18 Jan 20:41:26 AEDT 2018
stack:               cflinuxfs2
buildpack:           client-certificate-mapper=1.4.0_RELEASE container-security-provider=1.11.0_RELEASE java-buildpack=v4.7.1-offline-https://github.com/cloudfoundry/java-buildpack.git#6a3361a
                     java-main java-opts java-security jvmkill-agent=1... (no decorators apply)

     state     since                  cpu    memory           disk           details
#0   running   2018-01-18T09:44:07Z   0.4%   224.8M of 756M   137.5M of 1G
#1   running   2018-01-18T09:44:13Z   0.8%   205M of 756M     137.5M of 1G
#2   running   2018-01-18T09:44:06Z   0.7%   221.1M of 756M   137.5M of 1G

Now lets simply access our application a few times using the "/" end point and verify we are accessing different application containers via round robin routing as per GoRouter

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 09:58:10 GMT
Set-Cookie: dtCookie=6$B570EBB532CD9D8DAA2BCAE14C4277FC|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: 336ba633-685b-4235-467d-b9833a9e6435

{
    "index": "2",
    "ip": "192.168.16.34"
}

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 09:58:15 GMT
Set-Cookie: dtCookie=5$3389B3DFBAD936D68CBAF30657653465|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: aa74e093-9031-4df5-73a5-bc9f1741a942

{
    "index": "1",
    "ip": "192.168.16.32"
}

Now we can request access to just the container with application index "1" as follows

1. First get the Application GUID as shown below

pasapicella@pas-macbook:~$ cf app pas-pcf-routingdemo --guid
5bdf2f08-34a5-402f-b7cb-f29c81d171e0

2. Now lets invoke a call to the application and set the HEADER required to instruct GoRouter to target a specific application index

eg: curl app.example.com -H "X-CF-APP-INSTANCE":"YOUR-APP-GUID:YOUR-INSTANCE-INDEX"

Example below is using HTTPie 

Accessing Instance 1

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/ "X-CF-APP-INSTANCE":"5bdf2f08-34a5-402f-b7cb-f29c81d171e0:1"
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 10:20:31 GMT
Set-Cookie: dtCookie=5$FD08A5C88469AF379C8AD3F36FA7984B|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: cb19b960-713a-49d0-4529-a0766a8880a7

{
    "index": "1",
    "ip": "192.168.16.32"
}

Accessing Instance 2 

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/ "X-CF-APP-INSTANCE":"5bdf2f08-34a5-402f-b7cb-f29c81d171e0:2"
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 10:21:09 GMT
Set-Cookie: dtCookie=7$53957A744D473BB024EB1FF4F0CD60A9|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: 33cc7922-9b43-4182-5c36-13eee42a9919

{
    "index": "2",
    "ip": "192.168.16.34"
}

More Information

https://docs.cloudfoundry.org/concepts/http-routing.html#http-headers

Friday, 29 December 2017

Verifying PCF 2.0 with PAS small footprint with bosh CLI

After installing PCF 2.0 here is how you can verify your installation using the new bosh2 CLI. In this example I use "bosh2" BUT with PCF 2.0 you can actually use "bosh". bosh2 v2 existed for a while in PCF 1.12 and some previous versions while we left bosh v1

1. SSH into your ops manager VM as shown below, in this example we using GCP

https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html#ssh

2. Create an alias for your ENV as shown below

Note: You will need the bosh director IP address which you can obtain using

  https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html#gather

ubuntu@opsman-pcf:~$ bosh2 alias-env gcp -e y.y.y.y --ca-cert /var/tempest/workspaces/default/root_ca_certificate
Using environment 'y.y.y.y' as anonymous user

Name      p-bosh
UUID      3c886290-144f-4ec7-86dd-b7586b98dc3b
Version   264.4.0 (00000000)
CPI       google_cpi
Features  compiled_package_cache: disabled
          config_server: enabled
          dns: disabled
          snapshots: disabled
User      (not logged in)

Succeeded

3. Log in to the BOSH Director with UAA

Note: You will need the username / password for the bosh director which you can obtain as follows

  https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html#gather

ubuntu@opsman-pcf:~$ bosh2 -e gcp log-in
Email (): director
Password ():

Successfully authenticated with UAA

Succeeded

4. View all the VM's managed by BOSH as follows

ubuntu@opsman-pcf:~/scripts$ bosh2 -e gcp vms --column=Instance --column="Process State" --column=AZ --column="VM Type"
Using environment 'y.y.y.y' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Task 65. Done

Deployment 'cf-adee3657c74c7b9a8e35'

Instance                                             Process State  AZ                      VM Type
backup-prepare/996340c7-4114-472e-b660-a5e353493fa4  running        australia-southeast1-a  micro
blobstore/cdd6fc8d-25c9-4cfb-9908-89eb0164fb80       running        australia-southeast1-a  medium.mem
compute/2dfcc046-c16a-4a36-9170-ef70d1881818         running        australia-southeast1-a  xlarge.disk
control/2f3d0bc6-9a2d-4c08-9ccc-a88bad6382a3         running        australia-southeast1-a  xlarge
database/da60f0e7-b8e3-4f8d-945d-306b267ac161        running        australia-southeast1-a  large.disk
mysql_monitor/a88331c4-1659-4fe4-b8e9-89ce4bf092fd   running        australia-southeast1-a  micro
router/276e308e-a476-4c8d-9555-21623dada492          running        australia-southeast1-a  micro

7 vms

Succeeded

** Few other examples **

- View all the deployments, in this example we just have the PAS small footprint tile installed so it only exists and no other bosh managed tiles xist

ubuntu@opsman-pcf:~/scripts$ bosh2 -e gcp deployments --column=name
Using environment 'y.y.y.y' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Name
cf-adee3657c74c7b9a8e35

1 deployments

Succeeded

- Run cloud check to check for issues

ubuntu@opsman-pcf:~/scripts$ bosh2 -e gcp -d cf-adee3657c74c7b9a8e35 cloud-check
Using environment 'y.y.y.y' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Using deployment 'cf-adee3657c74c7b9a8e35'

Task 66

Task 66 | 04:20:52 | Scanning 7 VMs: Checking VM states (00:00:06)
Task 66 | 04:20:58 | Scanning 7 VMs: 7 OK, 0 unresponsive, 0 missing, 0 unbound (00:00:00)
Task 66 | 04:20:58 | Scanning 3 persistent disks: Looking for inactive disks (00:00:01)
Task 66 | 04:20:59 | Scanning 3 persistent disks: 3 OK, 0 missing, 0 inactive, 0 mount-info mismatch (00:00:00)

Task 66 Started  Fri Dec 29 04:20:52 UTC 2017
Task 66 Finished Fri Dec 29 04:20:59 UTC 2017
Task 66 Duration 00:00:07
Task 66 done

#  Type  Description

0 problems

Succeeded

More Information

https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html