fabric-samples/test-network-k8s/docs/KUBERNETES.md

# Kubernetes 

To get started with the Kube test network, you will need access to a Kubernetes cluster.

## TL/DR : 

```shell
$ ./network kind 
Initializing KIND cluster "kind":
✅ - Pulling docker images for Fabric 2.3.2 ...
✅ - Creating cluster "kind" ...
✅ - Launching Nginx ingress controller ...
✅ - Launching container registry "kind-registry" at localhost:5000 ...
🏁 - Cluster is ready.
```

and :
```shell
$ ./network unkind 
Deleting cluster "kind":
☠️  - Deleting KIND cluster kind ...
🏁 - Cluster is gone.
```


## Kube Context: 

For illustration purposes, this project attempts in all cases to _keep it simple_ as the
general rule.  By default, we will rely on `kind` ([Kubernetes IN Docker](https://kind.sigs.k8s.io))
as a mechanism to quickly spin up ephemeral, short-lived clusters for development and 
illustration.

To maximize portability across revisions, vendor distributions, hardware profiles, and 
network topologies, this project relies _exclusively_ on scripted interaction with the 
Kube API controller to reflect updates in a remote cluster.  While this may not be the 
ideal technique for managing production workloads, the objective of this guide is to provide 
clarity on the nuances of Fabric / Kubernetes deployments, rather than an opinionated 
perspective on state of the art techniques for cloud Dev/Ops.  Targeting 
the core Kube APIs means that there is a good chance that the systems will work "as-is" 
simply by setting the kubectl context to reference a cloud-native cluster (e.g. OCP, IKS, 
AWS, etc.)

If you don't have access to an existing cluster, or want to set up a short-lived cluster 
for development, testing, or CI, you can create a new cluster with:

```shell
$ ./network kind
```
or: 
```shell
$ kind create cluster 
```

By default, `kind` will set the current Kube context to reference the new cluster.  Any 
interaction with `kubectl` (or kube-context aware SDKs) will inherit the current context. 

```shell
$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:55346
CoreDNS is running at https://127.0.0.1:55346/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```

When you are done with the cluster, tear it down with:
```shell
$ ./network unkind 
```
or: 
```shell
$ kind delete cluster 
```

## Test Network Structure

To emulate a more realistic example of multi-party collaboration, the test network 
forms a blockchain consensus group spanning three virtual organizations.  Access to the 
blockchain is entirely constrained to Kubernetes private networks, and consuming applications 
make use of a Kube ingress controller for external visibility.

In k8s terms: 

- The blockchain is contained within a single Kubernetes `Cluster`. 
- Blockchain services (nodes, orderers, chaincode, etc.) reside within a single `Namespace`.
- Each organization maintains a distinct, independent `PersistentVolumeClaim` for TLS certificates,
  local MSP, private data, and transaction ledgers.
- Smart Contracts rely exclusively on the [Chaincode-as-a-Service](link) and [External Builder](link)
  patterns, running in the cluster as Kube `Deployments` with companion `Services`. 
- An HTTP(s) `Ingress` and companion gateway application is required for external access to the blockchain. 

When running the test network locally, the `./network kind` bootstrap will configure the system with 
an [Nginx ingress controller](link), a private [Container Registry](link), and persistent volumes / claims for 
host-local organization storage. 

Behind the scenes, `./network kind` is running: 

```shell
# Create the KIND cluster and nginx ingress controller bound to :80 and :443 
kind create cluster --name ${TEST_NETWORK_CLUSTER_NAME:-kind} --config scripts/kind-config.yaml

# Create the Kube namespace 
kubectl create namespace ${TEST_NETWORK_NAMESPACE:-test-network}

# Create host persistent volumes (tied the kind-control-plane docker image lifetime)
kubectl create -f kube/pv-fabric-org0.yaml
kubectl create -f kube/pv-fabric-org1.yaml
kubectl create -f kube/pv-fabric-org2.yaml

# Create persistent volume claims binding to the host (docker) volumes 
kubectl -n $NS create -f kube/pvc-fabric-org0.yaml 
kubectl -n $NS create -f kube/pvc-fabric-org1.yaml 
kubectl -n $NS create -f kube/pvc-fabric-org2.yaml 
```

## Container Registry

The [kube yaml descriptors](../kube) generally rely on the public Fabric images maintained at the public 
Docker and GitHub container registries.  For casual usage, the test network will bootstrap and launch CAs, 
peers, orderers, chaincode, and sample applications without any additional configuration.

While public images are made available for pre-canned samples, there will undoubtedly be cases 
where you would like to build custom chaincode, gateway client applications, or custom builds of core 
Fabric binaries without uploading your code to a public registry.  For this purpose, the Kube test 
network includes a [Local Registry](https://kind.sigs.k8s.io/docs/user/local-registry/) available for 
you to _quickly_ deploy custom images directly into the cluster without uploading your code to the 
Internet.

By default, the [kind.sh](../scripts/kind.sh) bootstrap will configure and link up a local container 
registry running at `localhost:5000/`.  Images pushed to this registry will be immediately available 
to Pods deployed to the local cluster.  

For dev/test/CI based flows using an external registry, the traditional Kubernetes practice of 
[Adding ImagePullSecrets to a service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account) 
still applies.

In some environments, KIND may encounter issues loading the Fabric docker images from the public container 
registries.  In addition, for Fabric development it can be advantageous to work with Docker images built 
locally, bypassing the public images entirely.  For these scenarios, images may also be [directly loaded](https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster)
into the KIND image plane, bypassing the container registry.

The `./network` script supports these additional modes via: 

1.  For network-constrained environments, pull all images to the local docker cache and load to KIND: 
```shell
export TEST_NETWORK_STAGE_DOCKER_IMAGES=true 

./network kind 
./network up  
```

2.  For alternate registries (e.g. local or Fabric CI/CD builds): 
```shell
./network kind 

export TEST_NETWORK_FABRIC_CONTAINER_REGISTRY=hyperledger-fabric.jfrog.io
export TEST_NETWORK_FABRIC_VERSION=amd64-latest 
export TEST_NETWORK_FABRIC_CA_VERSION=amd64-latest

./network up  
```

3.  For working with Fabric images built locally: 
```shell
./network kind 

make docker    # in hyperledger/fabric 

export TEST_NETWORK_FABRIC_VERSION=2.4.0

./network load-images 
./network up 
```

## Cloud Vendors 

While the test network primarily targets KIND clusters, the singular reliance on the Kube API plane 
means that it should also work without modification on any modern cloud-based or bare metal 
Kubernetes distribution.  While supporting the entire ecosystem of cloud vendors is not in scope 
for this sample project, we'd love to hear feedback, success stories, or bugs related to applying the 
test network to additional platforms.

In general, at a high-level the steps required to port the test network to ANY kube vendor are: 

- Configure an HTTP `Ingress` for access to any gateway, REST, or companion blockchain applications.
- Register `PersistentVolumeClaims` for each of the organizations in the test network. 
- Create a `Namespace` for each instance of the test network.
- Upload your chaincode, gateway clients, and application logic to an external Container Registry. 
- Run with a `ServiceAccount` and role bindings suitable for creating `Pods`, `Deployments`, and `Services`. 

Example configurations for common cloud vendors: 

### IKS 
### OCP 
### AWS 
### Azure 

## Next : [Fabric Certificate Authorities](CA.md)