Kubernetes机器学习平台搭建构建企业级ML训练环境一、机器学习平台概述Kubernetes机器学习平台是基于K8s构建的ML训练和部署基础设施支持数据科学家进行模型训练、验证和部署。1.1 ML平台架构┌─────────────────────────┐ │ 用户界面 │ │ (Jupyter/TensorBoard) │ └───────────┬─────────────┘ │ ┌─────────────────────────┼─────────────────────────┐ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ 训练调度器 │ │ 模型仓库 │ │ 数据存储 │ │ (Kubeflow) │ │ (MLflow) │ │ (MinIO) │ └───────────────┘ └───────────────┘ └───────────────┘ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ GPU节点池 │ │ CPU节点池 │ │ 存储集群 │ │ (训练任务) │ │ (预处理) │ │ (数据/模型) │ └───────────────┘ └───────────────┘ └───────────────┘1.2 核心组件组件功能工具训练调度管理训练任务Kubeflow、Argo Workflows模型管理模型版本控制MLflow、DVC数据存储数据集管理MinIO、PV/PVC资源管理GPU/CPU调度Kubernetes调度器可视化实验追踪TensorBoard、Weights Biases二、Kubeflow部署2.1 Kubeflow安装# 安装Kubeflow export KUBEFLOW_RELEASE_VERSIONv1.8.0 export KUSTOMIZE_VERSIONv5.0.1 git clone https://github.com/kubeflow/manifests.git cd manifests git checkout ${KUBEFLOW_RELEASE_VERSION} # 部署Kubeflow while ! kustomize build example | kubectl apply -f -; do echo Retrying...; sleep 10; done2.2 Kubeflow Pipeline配置apiVersion: kubeflow.org/v1 kind: Pipeline metadata: name: ml-pipeline spec: pipelineSpec: tasks: - name: preprocess taskSpec: podSpec: containers: - name: preprocess image: preprocess:latest command: [python, preprocess.py] - name: train taskSpec: podSpec: containers: - name: train image: train:latest command: [python, train.py] resources: limits: nvidia.com/gpu: 1 dependencies: - preprocess三、MLflow配置3.1 MLflow部署apiVersion: apps/v1 kind: Deployment metadata: name: mlflow namespace: mlflow spec: replicas: 1 selector: matchLabels: app: mlflow template: metadata: labels: app: mlflow spec: containers: - name: mlflow image: mlflow:latest ports: - containerPort: 5000 env: - name: MLFLOW_S3_ENDPOINT_URL value: http://minio:9000 - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: minio-creds key: accesskey - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: minio-creds key: secretkey command: - mlflow - server - --host0.0.0.0 - --port5000 - --backend-store-uripostgresql://mlflow:passwordpostgres/mlflow - --default-artifact-roots3://mlflow/3.2 MLflow模型注册import mlflow import mlflow.sklearn mlflow.set_tracking_uri(http://mlflow:5000) with mlflow.start_run(): # 训练模型 model train_model() # 记录参数 mlflow.log_param(learning_rate, 0.01) # 记录指标 mlflow.log_metric(accuracy, 0.95) # 保存模型 mlflow.sklearn.log_model(model, model) # 注册模型 mlflow.register_model( runs:/{}/model.format(mlflow.active_run().info.run_id), my-model )四、GPU资源管理4.1 GPU节点配置apiVersion: v1 kind: Node metadata: name: gpu-node-01 labels: nvidia.com/gpu.present: true node-role.kubernetes.io/gpu: spec: taints: - key: nvidia.com/gpu value: true effect: NoSchedule4.2 GPU资源请求apiVersion: v1 kind: Pod metadata: name: gpu-training-pod spec: tolerations: - key: nvidia.com/gpu operator: Equal value: true effect: NoSchedule containers: - name: training image: tensorflow/tensorflow:latest-gpu command: [python, train.py] resources: limits: nvidia.com/gpu: 2 memory: 32Gi cpu: 8 requests: nvidia.com/gpu: 2 memory: 16Gi cpu: 4五、数据存储配置5.1 MinIO部署apiVersion: apps/v1 kind: StatefulSet metadata: name: minio namespace: minio spec: serviceName: minio replicas: 4 selector: matchLabels: app: minio template: metadata: labels: app: minio spec: containers: - name: minio image: minio/minio:latest ports: - containerPort: 9000 command: - minio - server - /data - --console-address - :9001 volumeMounts: - name: data mountPath: /data env: - name: MINIO_ROOT_USER valueFrom: secretKeyRef: name: minio-creds key: accesskey - name: MINIO_ROOT_PASSWORD valueFrom: secretKeyRef: name: minio-creds key: secretkey volumeClaimTemplates: - metadata: name: data spec: accessModes: [ReadWriteOnce] resources: requests: storage: 100Gi5.2 PVC配置apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ml-data namespace: ml spec: accessModes: - ReadWriteMany resources: requests: storage: 500Gi storageClassName: nfs-storage六、JupyterHub部署6.1 JupyterHub配置apiVersion: hub.jupyter.org/v1 kind: Hub metadata: name: jupyterhub namespace: jupyterhub spec: image: name: jupyterhub/k8s-hub tag: 2.0.0 proxy: secretToken: secret-token auth: type: github github: clientId: client-id clientSecret: client-secret callbackUrl: https://jupyter.example.com/hub/oauth_callback singleuser: image: name: jupyter/scipy-notebook tag: latest storage: type: persistent-claim capacity: 10Gi6.2 用户配置apiVersion: hub.jupyter.org/v1 kind: User metadata: name: datascientist namespace: jupyterhub spec: profile: displayName: Data Scientist admin: false server: resources: limits: cpu: 4 memory: 16Gi requests: cpu: 2 memory: 8Gi七、TensorBoard配置7.1 TensorBoard部署apiVersion: v1 kind: Service metadata: name: tensorboard namespace: ml spec: type: ClusterIP selector: app: tensorboard ports: - port: 6006 targetPort: 6006 --- apiVersion: apps/v1 kind: Deployment metadata: name: tensorboard namespace: ml spec: replicas: 1 selector: matchLabels: app: tensorboard template: metadata: labels: app: tensorboard spec: containers: - name: tensorboard image: tensorflow/tensorflow:latest command: - tensorboard - --logdir/logs - --host0.0.0.0 ports: - containerPort: 6006 volumeMounts: - name: logs mountPath: /logs volumes: - name: logs persistentVolumeClaim: claimName: tensorboard-logs八、模型部署8.1 TensorFlow ServingapiVersion: v1 kind: Service metadata: name: tf-serving namespace: ml spec: type: ClusterIP selector: app: tf-serving ports: - port: 8501 targetPort: 8501 --- apiVersion: apps/v1 kind: Deployment metadata: name: tf-serving namespace: ml spec: replicas: 3 selector: matchLabels: app: tf-serving template: metadata: labels: app: tf-serving spec: containers: - name: tf-serving image: tensorflow/serving:latest ports: - containerPort: 8500 - containerPort: 8501 args: - --model_namemy-model - --model_base_paths3://models/my-model env: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: minio-creds key: accesskey - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: minio-creds key: secretkey - name: S3_ENDPOINT value: http://minio:90008.2 gRPC推理服务apiVersion: v1 kind: Service metadata: name: model-service namespace: ml spec: type: ClusterIP selector: app: model-service ports: - port: 9000 targetPort: 9000 name: grpc - port: 8080 targetPort: 8080 name: http九、监控与日志9.1 训练指标监控apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ml-monitor namespace: monitoring spec: selector: matchLabels: app: ml-exporter endpoints: - port: metrics interval: 15s9.2 资源使用监控apiVersion: v1 kind: ConfigMap metadata: name: ml-metrics-config namespace: monitoring data: prometheus.rules: | groups: - name: ml.rules rules: - record: ml_training_duration_seconds expr: sum(rate(kube_pod_running_duration_seconds{apptraining}[5m])) - record: ml_gpu_utilization expr: sum(nvidia_gpu_utilization{jobnvidia-dcgm-exporter})十、总结Kubernetes机器学习平台搭建需要考虑训练调度使用Kubeflow管理ML工作流模型管理使用MLflow进行模型版本控制GPU资源配置GPU节点池和资源调度数据存储部署MinIO管理数据集开发环境使用JupyterHub提供交互式开发可视化配置TensorBoard进行实验追踪模型部署使用TensorFlow Serving部署模型监控告警建立训练指标和资源使用监控建议根据团队规模和业务需求选择合适的组件构建高效的ML平台。参考资料Kubeflow官方文档MLflow文档JupyterHub Kubernetes文档