Enabled SCTP in Kubernetes Cluster

When we are developing cloud services, we will need SCTP sometimes, but we don’t have SCTP enabled on kubernetes in default, this tutorial go through with how to boot a kubernetes cluster with SCTP enabled, and onboard a SCTP server for testing.

Environment

  • OS: Ubuntu 18.04 LTS
  • Kubernetes: 1.13.1

Composing Service with SCTP

Here comes a example of how to write a Service with SCTP. Also, you can find a example helm-chart at here: aweimeow/sctp-server, this example packed StackOverflow - Python SCTP Server Side Module‘s source code into the container image.

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: Service
metadata:
name: sctp
spec:
selector:
app: sctp
ports:
- protocol: SCTP
port: 9999
targetPort: 30001

Boot Kubernetes cluster with SCTP enabled

If we are going to make kubernetes support with SCTP, we can use SCTPSupport=true to enable SCTP function in cluster. And I set the networking parameters in the sametime, networking/podSubnet corresponds to --pod-network-cidr parameter when we boot the cluster by arguments.

1
2
3
4
5
6
7
8
9
10
11
# config.yaml
---
apiVersion: kubeadm.k8s.io/v1alpha3
kubernetesVersion: v1.13.1
kind: ClusterConfiguration
apiServerExtraArgs:
feature-gates: SCTPSupport=true
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12

And using this command to start cluster by assigning configuration file, you can join other nodes into this cluster by command’s output.

1
$ sudo kubeadm init --config config.yaml

Deploying Service into cluster

First, we need to install tiller pod into this cluster, but you may need to have tiller account have cluster-admin privilege, so helm can use User tiller to deploy services and pods on this cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system

Then you can install tiller pod on cluster, and deploy chart to this cluster.

1
2
3
$ helm init --service-account tiller
$ git clone https://github.com/aweimeow/sctp-server
$ helm install -n sctp sctp-server

Check SCTP functionality with cluster

First of all, let us check our service is deployed successfully.

1
2
3
4
5
6
7
[email protected]:~$ kubectl -n sctp get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sctp NodePort 10.110.200.110 <none> 9999:30001/SCTP 18s

[email protected]:~$ kubectl -n sctp get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
sctp-7cb7f5bfdc-85rxj 1/1 Running 0 2m33s 10.244.3.109 node2 <none> <none>

Because our pod is deployed on node2, so we can verify SCTP by following command:

1
2
3
4
[email protected]:~$ ncat --sctp node2 30001
Howdy! What's your name?
aweimeow
Thanks for calling, aweimeow. Bye, now.

You can also check SCTP traffic by tcpdump.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[email protected]:~$ sudo tcpdump -env sctp
tcpdump: listening on enp8s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:36:58.965582 a4:bf:01:04:8c:c8 > a4:bf:01:04:8c:64, ethertype IPv4 (0x0800), length 122: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto SCTP (132), length 108)
10.90.0.234.52509 > 10.90.0.213.30001: sctp (1) [INIT] [init tag: 288371897] [rwnd: 106496] [OS: 10] [MIS: 65535] [init TSN: 3640648681]
21:36:58.965821 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 338: (tos 0x2,ECT(0), ttl 63, id 0, offset 0, flags [DF], proto SCTP (132), length 324)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [INIT ACK] [init tag: 3994388771] [rwnd: 106496] [OS: 10] [MIS: 10] [init TSN: 1048664571]
21:36:58.965887 a4:bf:01:04:8c:c8 > a4:bf:01:04:8c:64, ethertype IPv4 (0x0800), length 310: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto SCTP (132), length 296)
10.90.0.234.52509 > 10.90.0.213.30001: sctp (1) [COOKIE ECHO]
21:36:58.966012 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 60: (tos 0x2,ECT(0), ttl 63, id 0, offset 0, flags [DF], proto SCTP (132), length 36)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [COOKIE ACK]
21:36:58.966634 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 90: (tos 0x2,ECT(0), ttl 63, id 32700, offset 0, flags [DF], proto SCTP (132), length 76)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [DATA] (B)(E) [TSN: 1048664571] [SID: 0] [SSEQ 0] [PPID 0x0]
21:36:58.966683 a4:bf:01:04:8c:c8 > a4:bf:01:04:8c:64, ethertype IPv4 (0x0800), length 62: (tos 0x2,ECT(0), ttl 64, id 1, offset 0, flags [DF], proto SCTP (132), length 48)
10.90.0.234.52509 > 10.90.0.213.30001: sctp (1) [SACK] [cum ack 1048664571] [a_rwnd 106471] [#gap acks 0] [#dup tsns 0]
21:37:00.579703 a4:bf:01:04:8c:c8 > a4:bf:01:04:8c:64, ethertype IPv4 (0x0800), length 70: (tos 0x2,ECT(0), ttl 64, id 2, offset 0, flags [DF], proto SCTP (132), length 56)
10.90.0.234.52509 > 10.90.0.213.30001: sctp (1) [DATA] (B)(E) [TSN: 3640648681] [SID: 0] [SSEQ 0] [PPID 0x0]
21:37:00.579883 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 62: (tos 0x2,ECT(0), ttl 63, id 32701, offset 0, flags [DF], proto SCTP (132), length 48)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [SACK] [cum ack 3640648681] [a_rwnd 106490] [#gap acks 0] [#dup tsns 0]
21:37:00.579945 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 98: (tos 0x2,ECT(0), ttl 63, id 32702, offset 0, flags [DF], proto SCTP (132), length 84)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [DATA] (B)(E) [TSN: 1048664572] [SID: 0] [SSEQ 1] [PPID 0x0]
21:37:00.785875 a4:bf:01:04:8c:c8 > a4:bf:01:04:8c:64, ethertype IPv4 (0x0800), length 62: (tos 0x2,ECT(0), ttl 64, id 3, offset 0, flags [DF], proto SCTP (132), length 48)
10.90.0.234.52509 > 10.90.0.213.30001: sctp (1) [SACK] [cum ack 1048664572] [a_rwnd 106496] [#gap acks 0] [#dup tsns 0]
21:37:00.786017 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 60: (tos 0x2,ECT(0), ttl 63, id 32703, offset 0, flags [DF], proto SCTP (132), length 40)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [SHUTDOWN]
21:37:00.786061 a4:bf:01:04:8c:c8 > a4:bf:01:04:8c:64, ethertype IPv4 (0x0800), length 50: (tos 0x2,ECT(0), ttl 64, id 4, offset 0, flags [DF], proto SCTP (132), length 36)
10.90.0.234.52509 > 10.90.0.213.30001: sctp (1) [SHUTDOWN ACK]
21:37:00.786142 a4:bf:01:04:8c:64 > a4:bf:01:04:8c:c8, ethertype IPv4 (0x0800), length 60: (tos 0x2,ECT(0), ttl 63, id 32704, offset 0, flags [DF], proto SCTP (132), length 36)
10.90.0.213.30001 > 10.90.0.234.52509: sctp (1) [SHUTDOWN COMPLETE]