Update test-network-k8s to use two digit version (#668)

Instead of using the Fabric and Fabric CA three digit version (2.4.3),
utilize the two digit version (2.4).
Each time a Fabric/FabricCA release is pushed to dockerhub the two digit version tag is updated.
This approach simplifies maintenance so that scripts don't have to be updated for every
third digit release.

Signed-off-by: David Enyeart <enyeart@us.ibm.com>
This commit is contained in:
Dave Enyeart 2022-03-03 18:16:19 -05:00 committed by GitHub
parent b2d74ddbab
commit 67d3c65847
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 133 additions and 139 deletions

View file

@ -1,11 +1,11 @@
# Kubernetes
# Kubernetes
To get started with the Kube test network, you will need access to a Kubernetes cluster.
## TL/DR :
## TL/DR :
```shell
$ ./network kind
$ ./network kind
Initializing KIND cluster "kind":
✅ - Pulling docker images for Fabric 2.3.2 ...
✅ - Creating cluster "kind" ...
@ -16,43 +16,43 @@ Initializing KIND cluster "kind":
and :
```shell
$ ./network unkind
$ ./network unkind
Deleting cluster "kind":
☠️ - Deleting KIND cluster kind ...
🏁 - Cluster is gone.
```
## Kube Context:
## Kube Context:
For illustration purposes, this project attempts in all cases to _keep it simple_ as the
general rule. By default, we will rely on `kind` ([Kubernetes IN Docker](https://kind.sigs.k8s.io))
as a mechanism to quickly spin up ephemeral, short-lived clusters for development and
as a mechanism to quickly spin up ephemeral, short-lived clusters for development and
illustration.
To maximize portability across revisions, vendor distributions, hardware profiles, and
network topologies, this project relies _exclusively_ on scripted interaction with the
Kube API controller to reflect updates in a remote cluster. While this may not be the
ideal technique for managing production workloads, the objective of this guide is to provide
clarity on the nuances of Fabric / Kubernetes deployments, rather than an opinionated
perspective on state of the art techniques for cloud Dev/Ops. Targeting
the core Kube APIs means that there is a good chance that the systems will work "as-is"
simply by setting the kubectl context to reference a cloud-native cluster (e.g. OCP, IKS,
To maximize portability across revisions, vendor distributions, hardware profiles, and
network topologies, this project relies _exclusively_ on scripted interaction with the
Kube API controller to reflect updates in a remote cluster. While this may not be the
ideal technique for managing production workloads, the objective of this guide is to provide
clarity on the nuances of Fabric / Kubernetes deployments, rather than an opinionated
perspective on state of the art techniques for cloud Dev/Ops. Targeting
the core Kube APIs means that there is a good chance that the systems will work "as-is"
simply by setting the kubectl context to reference a cloud-native cluster (e.g. OCP, IKS,
AWS, etc.)
If you don't have access to an existing cluster, or want to set up a short-lived cluster
If you don't have access to an existing cluster, or want to set up a short-lived cluster
for development, testing, or CI, you can create a new cluster with:
```shell
$ ./network kind
```
or:
or:
```shell
$ kind create cluster
$ kind create cluster
```
By default, `kind` will set the current Kube context to reference the new cluster. Any
interaction with `kubectl` (or kube-context aware SDKs) will inherit the current context.
By default, `kind` will set the current Kube context to reference the new cluster. Any
interaction with `kubectl` (or kube-context aware SDKs) will inherit the current context.
```shell
$ kubectl cluster-info
@ -64,41 +64,41 @@ To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
When you are done with the cluster, tear it down with:
```shell
$ ./network unkind
$ ./network unkind
```
or:
or:
```shell
$ kind delete cluster
$ kind delete cluster
```
## Test Network Structure
To emulate a more realistic example of multi-party collaboration, the test network
forms a blockchain consensus group spanning three virtual organizations. Access to the
blockchain is entirely constrained to Kubernetes private networks, and consuming applications
To emulate a more realistic example of multi-party collaboration, the test network
forms a blockchain consensus group spanning three virtual organizations. Access to the
blockchain is entirely constrained to Kubernetes private networks, and consuming applications
make use of a Kube ingress controller for external visibility.
In k8s terms:
In k8s terms:
- The blockchain is contained within a single Kubernetes `Cluster`.
- The blockchain is contained within a single Kubernetes `Cluster`.
- Blockchain services (nodes, orderers, chaincode, etc.) reside within a single `Namespace`.
- Each organization maintains a distinct, independent `PersistentVolumeClaim` for TLS certificates,
local MSP, private data, and transaction ledgers.
- Smart Contracts rely exclusively on the [Chaincode-as-a-Service](link) and [External Builder](link)
patterns, running in the cluster as Kube `Deployments` with companion `Services`.
- An HTTP(s) `Ingress` and companion gateway application is required for external access to the blockchain.
patterns, running in the cluster as Kube `Deployments` with companion `Services`.
- An HTTP(s) `Ingress` and companion gateway application is required for external access to the blockchain.
When running the test network locally, the `./network kind` bootstrap will configure the system with
an [Nginx ingress controller](link), a private [Container Registry](link), and persistent volumes / claims for
host-local organization storage.
When running the test network locally, the `./network kind` bootstrap will configure the system with
an [Nginx ingress controller](link), a private [Container Registry](link), and persistent volumes / claims for
host-local organization storage.
Behind the scenes, `./network kind` is running:
Behind the scenes, `./network kind` is running:
```shell
# Create the KIND cluster and nginx ingress controller bound to :80 and :443
# Create the KIND cluster and nginx ingress controller bound to :80 and :443
kind create cluster --name ${TEST_NETWORK_CLUSTER_NAME:-kind} --config scripts/kind-config.yaml
# Create the Kube namespace
# Create the Kube namespace
kubectl create namespace ${TEST_NETWORK_NAMESPACE:-test-network}
# Create host persistent volumes (tied the kind-control-plane docker image lifetime)
@ -106,93 +106,92 @@ kubectl create -f kube/pv-fabric-org0.yaml
kubectl create -f kube/pv-fabric-org1.yaml
kubectl create -f kube/pv-fabric-org2.yaml
# Create persistent volume claims binding to the host (docker) volumes
kubectl -n $NS create -f kube/pvc-fabric-org0.yaml
kubectl -n $NS create -f kube/pvc-fabric-org1.yaml
kubectl -n $NS create -f kube/pvc-fabric-org2.yaml
# Create persistent volume claims binding to the host (docker) volumes
kubectl -n $NS create -f kube/pvc-fabric-org0.yaml
kubectl -n $NS create -f kube/pvc-fabric-org1.yaml
kubectl -n $NS create -f kube/pvc-fabric-org2.yaml
```
## Container Registry
The [kube yaml descriptors](../kube) generally rely on the public Fabric images maintained at the public
Docker and GitHub container registries. For casual usage, the test network will bootstrap and launch CAs,
The [kube yaml descriptors](../kube) generally rely on the public Fabric images maintained at the public
Docker and GitHub container registries. For casual usage, the test network will bootstrap and launch CAs,
peers, orderers, chaincode, and sample applications without any additional configuration.
While public images are made available for pre-canned samples, there will undoubtedly be cases
where you would like to build custom chaincode, gateway client applications, or custom builds of core
Fabric binaries without uploading your code to a public registry. For this purpose, the Kube test
network includes a [Local Registry](https://kind.sigs.k8s.io/docs/user/local-registry/) available for
you to _quickly_ deploy custom images directly into the cluster without uploading your code to the
While public images are made available for pre-canned samples, there will undoubtedly be cases
where you would like to build custom chaincode, gateway client applications, or custom builds of core
Fabric binaries without uploading your code to a public registry. For this purpose, the Kube test
network includes a [Local Registry](https://kind.sigs.k8s.io/docs/user/local-registry/) available for
you to _quickly_ deploy custom images directly into the cluster without uploading your code to the
Internet.
By default, the [kind.sh](../scripts/kind.sh) bootstrap will configure and link up a local container
registry running at `localhost:5000/`. Images pushed to this registry will be immediately available
By default, the [kind.sh](../scripts/kind.sh) bootstrap will configure and link up a local container
registry running at `localhost:5000/`. Images pushed to this registry will be immediately available
to Pods deployed to the local cluster.
For dev/test/CI based flows using an external registry, the traditional Kubernetes practice of
[Adding ImagePullSecrets to a service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account)
For dev/test/CI based flows using an external registry, the traditional Kubernetes practice of
[Adding ImagePullSecrets to a service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account)
still applies.
In some environments, KIND may encounter issues loading the Fabric docker images from the public container
registries. In addition, for Fabric development it can be advantageous to work with Docker images built
In some environments, KIND may encounter issues loading the Fabric docker images from the public container
registries. In addition, for Fabric development it can be advantageous to work with Docker images built
locally, bypassing the public images entirely. For these scenarios, images may also be [directly loaded](https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster)
into the KIND image plane, bypassing the container registry.
The `./network` script supports these additional modes via:
The `./network` script supports these additional modes via:
1. For network-constrained environments, pull all images to the local docker cache and load to KIND:
1. For network-constrained environments, pull all images to the local docker cache and load to KIND:
```shell
export TEST_NETWORK_STAGE_DOCKER_IMAGES=true
export TEST_NETWORK_STAGE_DOCKER_IMAGES=true
./network kind
./network kind
./network up
```
2. For alternate registries (e.g. local or Fabric CI/CD builds):
2. For alternate registries (e.g. local or Fabric CI/CD builds):
```shell
./network kind
./network kind
export TEST_NETWORK_FABRIC_CONTAINER_REGISTRY=hyperledger-fabric.jfrog.io
export TEST_NETWORK_FABRIC_VERSION=amd64-latest
export TEST_NETWORK_FABRIC_VERSION=amd64-latest
export TEST_NETWORK_FABRIC_CA_VERSION=amd64-latest
./network up
```
3. For working with Fabric images built locally:
3. For working with Fabric images built locally:
```shell
./network kind
./network kind
make docker # in hyperledger/fabric
make docker # in hyperledger/fabric
export TEST_NETWORK_FABRIC_VERSION=2.4.0
./network load-images
./network up
./network load-images
./network up
```
## Cloud Vendors
## Cloud Vendors
While the test network primarily targets KIND clusters, the singular reliance on the Kube API plane
means that it should also work without modification on any modern cloud-based or bare metal
Kubernetes distribution. While supporting the entire ecosystem of cloud vendors is not in scope
for this sample project, we'd love to hear feedback, success stories, or bugs related to applying the
While the test network primarily targets KIND clusters, the singular reliance on the Kube API plane
means that it should also work without modification on any modern cloud-based or bare metal
Kubernetes distribution. While supporting the entire ecosystem of cloud vendors is not in scope
for this sample project, we'd love to hear feedback, success stories, or bugs related to applying the
test network to additional platforms.
In general, at a high-level the steps required to port the test network to ANY kube vendor are:
In general, at a high-level the steps required to port the test network to ANY kube vendor are:
- Configure an HTTP `Ingress` for access to any gateway, REST, or companion blockchain applications.
- Register `PersistentVolumeClaims` for each of the organizations in the test network.
- Register `PersistentVolumeClaims` for each of the organizations in the test network.
- Create a `Namespace` for each instance of the test network.
- Upload your chaincode, gateway clients, and application logic to an external Container Registry.
- Run with a `ServiceAccount` and role bindings suitable for creating `Pods`, `Deployments`, and `Services`.
- Upload your chaincode, gateway clients, and application logic to an external Container Registry.
- Run with a `ServiceAccount` and role bindings suitable for creating `Pods`, `Deployments`, and `Services`.
Example configurations for common cloud vendors:
Example configurations for common cloud vendors:
### IKS
### OCP
### AWS
### Azure
### IKS
### OCP
### AWS
### Azure
## Next : [Fabric Certificate Authorities](CA.md)

View file

@ -1,47 +1,47 @@
## Network Overview
After we have set up a series of TLS and ECert CA services, we'll use the CAs to generate
[Local MSP](https://hyperledger-fabric.readthedocs.io/en/latest/membership/membership.html#local-msps) structures for
After we have set up a series of TLS and ECert CA services, we'll use the CAs to generate
[Local MSP](https://hyperledger-fabric.readthedocs.io/en/latest/membership/membership.html#local-msps) structures for
all of the nodes, using the local MSPs to launch our network peers and orderers.
### TL/DR :
### TL/DR :
```shell
./network up
...
./network up
...
✅ - Creating local node MSP ...
✅ - Launching orderers ...
✅ - Launching peers ...
🏁 - Network is ready.
```
## Fabric MSP Context
## Fabric MSP Context
Before we launch the network peers and orderers, each node in the network needs to have available:
Before we launch the network peers and orderers, each node in the network needs to have available:
- TLS Root Certificates for all organizations in the network
- TLS Certificates and Signing Keys for SSL server/hostname verification of the network node
- Enrollment Certificates validating the network node identity (local MSP)
- Enrollment Certificates for an `Admin` identity / role for the organization.
- Enrollment Certificates for an `Admin` identity / role for the organization.
In order to create the local node MSP, we must first register and enroll the node identities with the ECert CAs, and
then organize the TLS and MSP certificates into a location suitable for launching the network services.
The key steps in this process are:
The key steps in this process are:
- [Registering and enrolling identities with a CA](https://hyperledger-fabric-ca.readthedocs.io/en/latest/deployguide/use_CA.html#registering-and-enrolling-identities-with-a-ca)
- [Create the local MSP of a node](https://hyperledger-fabric-ca.readthedocs.io/en/latest/deployguide/use_CA.html#create-the-local-msp-of-a-node)
In the test network, each organization includes a function that wraps the registration, enrollment, and MSP aggregation
into a series of fabric-ca-client calls. [The script](../scripts/test_network.sh) will be executed directly on the
org's ECert CA pod, with access to the persistent volume for storage of the MSP and TLS certificates. While this is
largely boilerplate scripting, the process is straightforward: For each node in the network, we'll use the CAs to
generate TLS+MSP certificates, bundling into an MSP with a `config.yaml` specifying the fabric roles associated with
the target usage in the network.
In the test network, each organization includes a function that wraps the registration, enrollment, and MSP aggregation
into a series of fabric-ca-client calls. [The script](../scripts/test_network.sh) will be executed directly on the
org's ECert CA pod, with access to the persistent volume for storage of the MSP and TLS certificates. While this is
largely boilerplate scripting, the process is straightforward: For each node in the network, we'll use the CAs to
generate TLS+MSP certificates, bundling into an MSP with a `config.yaml` specifying the fabric roles associated with
the target usage in the network.
For example, the ordering organization sets up the node local MSP with:
For example, the ordering organization sets up the node local MSP with:
```shell
# Each identity in the network needs a registration and enrollment.
fabric-ca-client register --id.name org0-orderer1 --id.secret ordererpw --id.type orderer --url https://org0-ca --mspdir $FABRIC_CA_CLIENT_HOME/org0-ca/rcaadmin/msp
@ -75,33 +75,33 @@ cp /var/hyperledger/fabric/organizations/ordererOrganizations/org0.example.com/o
```
## External Chaincode Builders
## External Chaincode Builders
Running Fabric in Kubernetes places some unique constraints on the Chaincode lifecycle:
Running Fabric in Kubernetes places some unique constraints on the Chaincode lifecycle:
- Many cloud-native vendors rely on [containerd.io](https://containerd.io) to manage the lifecycle of containers
within a cluster. By contrast, Fabric assumes the presence of a Docker daemon to compile and launch chaincode
containers. Without a local Docker daemon, Fabric's default chaincode pipeline is doomed!
- Many cloud-native vendors rely on [containerd.io](https://containerd.io) to manage the lifecycle of containers
within a cluster. By contrast, Fabric assumes the presence of a Docker daemon to compile and launch chaincode
containers. Without a local Docker daemon, Fabric's default chaincode pipeline is doomed!
- For security and operational concerns, it is a "non-starter" to run a docker daemon on Kubernetes worker nodes.
- For cloud-ready development, test, validation, CI/CD, and production practices, the use of the
[Chaincode as a Service](https://hyperledger-fabric.readthedocs.io/en/latest/cc_service.html) pattern provides a
- For cloud-ready development, test, validation, CI/CD, and production practices, the use of the
[Chaincode as a Service](https://hyperledger-fabric.readthedocs.io/en/latest/cc_service.html) pattern provides a
_vastly superior user experience_.
- Running Chaincode builds in Docker in Docker, running in Kubernetes in Docker is ... interesting. Let's
step back and _keep it simple_.
- Running Chaincode builds in Docker in Docker, running in Kubernetes in Docker is ... interesting. Let's
step back and _keep it simple_.
In the Kubernetes Test Network, we've incorporated the default `ccaas` external builder
(See [fabric #2884](https://github.com/hyperledger/fabric/issues/2884)) as an accelerator for working with
Chaincode-as-a-Service on Kubernetes. For `ccaas` smart contracts, when chaincode is installed on a peer, the
(See [fabric #2884](https://github.com/hyperledger/fabric/issues/2884)) as an accelerator for working with
Chaincode-as-a-Service on Kubernetes. For `ccaas` smart contracts, when chaincode is installed on a peer, the
external builder binaries will be invoked, bypassing the reliance on a local Docker daemon running in Kubernetes.
This configuration is accomplished by registering an external builder in the peer core.yaml:
This configuration is accomplished by registering an external builder in the peer core.yaml:
```yaml
externalBuilders:
@ -119,54 +119,51 @@ To trigger the external builder for a chaincode service, set the metadata.json `
}
```
- [x] Pro tip: Use the companion container registry at `localhost:5000` to deploy custom chaincode into the test network.
- [x] Pro tip: Deploy a chaincode with `address: host.docker.internal:9999` and attach your chaincode in a debugger.
- [x] Pro tip: Use the companion container registry at `localhost:5000` to deploy custom chaincode into the test network.
- [x] Pro tip: Deploy a chaincode with `address: host.docker.internal:9999` and attach your chaincode in a debugger.
## Starting Peers and Orderers
## Starting Peers and Orderers
```shell
✅ - Launching orderers ...
✅ - Launching peers ...
```
Once the local MSP structures for the network nodes have been created, the orderers and peers may be launched in the
namespace. System nodes will read base configuration files (orderer.yaml and core.yaml) from the organization
Once the local MSP structures for the network nodes have been created, the orderers and peers may be launched in the
namespace. System nodes will read base configuration files (orderer.yaml and core.yaml) from the organization
config folder, made available in Kubernetes as the `fabric-config${org}` config map.
Each orderer and peer creates one `Deployment`, `Pod`, and `Service` in the namespace. In addition, each org
defines an `orgN-peerM-config` `ConfigMap` with environment variable overrides replacing the default settings
in the core.yaml file. Note that each node's [environment](../kube/org1/org1-peer1.yaml) includes pointers to the
Each orderer and peer creates one `Deployment`, `Pod`, and `Service` in the namespace. In addition, each org
defines an `orgN-peerM-config` `ConfigMap` with environment variable overrides replacing the default settings
in the core.yaml file. Note that each node's [environment](../kube/org1/org1-peer1.yaml) includes pointers to the
node local MSP folders, certificates, and TLS signing keys that we generated above.
Note that the deployment yaml files include some basic template substitution and parameters. For simplicity and
clarity, we elected to use basic string substitution with sed/awk/bash/etc., rather than introduce a Kube template
Note that the deployment yaml files include some basic template substitution and parameters. For simplicity and
clarity, we elected to use basic string substitution with sed/awk/bash/etc., rather than introduce a Kube template
binding system (e.g. Helm, Kustomize, Kapitan, Ansible, etc.) for manipulating yaml templates:
```shell
cat kube/org0/org0-orderer1.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org0/org0-orderer2.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org0/org0-orderer3.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org0/org0-orderer1.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org0/org0-orderer2.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org0/org0-orderer3.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
# Wait for the orderers to completely start before launching the network peer nodes.
kubectl -n $NS rollout status deploy/org0-orderer1
kubectl -n $NS rollout status deploy/org0-orderer2
# Wait for the orderers to completely start before launching the network peer nodes.
kubectl -n $NS rollout status deploy/org0-orderer1
kubectl -n $NS rollout status deploy/org0-orderer2
kubectl -n $NS rollout status deploy/org0-orderer3
cat kube/org1/org1-peer1.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org1/org1-peer2.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org2/org2-peer1.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org2/org2-peer2.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org1/org1-peer1.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org1/org1-peer2.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org2/org2-peer1.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
cat kube/org2/org2-peer2.yaml | sed 's,{{FABRIC_VERSION}},'${FABRIC_VERSION}',g' | kubectl -n $NS -f -
```
- [x] Pro tip: Run an early-release Fabric build by setting `TEST_NETWORK_FABRIC_VERSION=2.4.0-beta`
## Next Steps :
After the peers and orderers have started, the Kube namespace includes pods, deployments, and service bindings for:
After the peers and orderers have started, the Kube namespace includes pods, deployments, and service bindings for:
- Org0 (org0.example.com):
- Org0 (org0.example.com):
- ECert Certificate Authority : https://org0-ca
- Orderer1 : grpcs://org0-orderer1
- Orderer2 : grpcs://org0-orderer2
@ -187,4 +184,3 @@ After the peers and orderers have started, the Kube namespace includes pods, dep
### Next : [Working With Channels](CHANNELS.md)

View file

@ -21,8 +21,8 @@ set -o errexit
# todo: refactor query/invoke to specify chaincode name (-n param)
CONTAINER_CLI=${CONTAINER_CLI:-docker}
FABRIC_VERSION=${TEST_NETWORK_FABRIC_VERSION:-2.4.3}
FABRIC_CA_VERSION=${TEST_NETWORK_FABRIC_CA_VERSION:-1.5.2}
FABRIC_VERSION=${TEST_NETWORK_FABRIC_VERSION:-2.4}
FABRIC_CA_VERSION=${TEST_NETWORK_FABRIC_CA_VERSION:-1.5}
FABRIC_CONTAINER_REGISTRY=${TEST_NETWORK_FABRIC_CONTAINER_REGISTRY:-hyperledger}
NETWORK_NAME=${TEST_NETWORK_NAME:-test-network}
CLUSTER_NAME=${TEST_NETWORK_KIND_CLUSTER_NAME:-kind}
@ -167,7 +167,7 @@ elif [ "${MODE}" == "chaincode" ]; then
elif [ "${ACTION}" == "query" ]; then
query_chaincode $@ >> ${LOG_FILE}
elif [ "${ACTION}" == "metadata" ]; then
query_chaincode_metadata >> ${LOG_FILE}
else
@ -192,4 +192,3 @@ else
print_help
exit 1
fi