常规

• 概述
• 单节点设置
• 集群设置
• 命令参考
• 文件系统 Shell
• 兼容性规范
• 下游开发人员指南
• 管理员兼容性指南
• 接口分类
• 文件系统规范

通用

• CLI 迷你集群
• 公平调用队列
• 本机库
• 代理用户
• 机架感知
• 安全模式
• 服务级别授权
• HTTP 身份验证
• 凭据提供程序 API
• Hadoop KMS
• 跟踪
• Unix Shell 指南
• 注册表

HDFS

• 架构
• 用户指南
• 命令参考
• 带有 QJM 的 NameNode HA
• 带有 NFS 的 NameNode HA
• 观察者 NameNode
• 联合
• ViewFs
• ViewFsOverloadScheme
• 快照
• 编辑器查看器
• 图像查看器
• 权限和 HDFS
• 配额和 HDFS
• libhdfs（C API）
• WebHDFS（REST API）
• HttpFS
• 短路本地读取
• 集中式缓存管理
• NFS 网关
• 滚动升级
• 扩展属性
• 透明加密
• 多归属
• 存储策略
• 内存存储支持
• 合成负载生成器
• 擦除编码
• 磁盘平衡器
• 升级域
• DataNode 管理员
• 路由器联合
• 提供的存储

MapReduce

• 教程
• 命令参考
• 与 1.x 的兼容性
• 加密的 Shuffle
• 可插拔的 Shuffle/排序
• 分布式缓存部署
• 支持 YARN 共享缓存

MapReduce REST API

• MR 应用程序主控
• MR 历史服务器

YARN

• 架构
• 命令参考
• 容量调度程序
• 公平调度程序
• ResourceManager 重启
• ResourceManager HA
• 资源模型
• 节点标签
• 节点属性
• Web 应用程序代理
• 时间线服务器
• 时间线服务 V.2
• 编写 YARN 应用程序
• YARN 应用程序安全
• NodeManager
• 在 Docker 容器中运行应用程序
• 在 runC 容器中运行应用程序
• 使用 CGroup
• 安全容器
• 预留系统
• 优雅地停用
• 机会容器
• YARN 联合
• 共享缓存
• 使用 GPU
• 使用 FPGA
• 放置约束
• YARN UI2

YARN REST API

• 简介
• 资源管理器
• 节点管理器
• 时间线服务器
• 时间线服务 V.2

YARN 服务

• 概述
• 快速入门
• 概念
• Yarn 服务 API
• 服务发现
• 系统服务

与 Hadoop 兼容的文件系统

• 阿里云 OSS
• 亚马逊 S3
• Azure Blob 存储
• Azure Data Lake 存储
• 腾讯 COS

身份验证

• 概述
• 示例
• 配置
• 构建

工具

• Hadoop 流式处理
• Hadoop 存档
• Hadoop 存档日志
• DistCp
• GridMix
• Rumen
• 资源估算器服务
• 调度器负载模拟器
• Hadoop 基准测试
• 测功机

参考

• 变更日志和发行说明
• Java API 文档
• Unix Shell API
• 指标

配置

• core-default.xml
• hdfs-default.xml
• hdfs-rbf-default.xml
• mapred-default.xml
• yarn-default.xml
• kms-default.xml
• httpfs-default.xml
• 已弃用的属性

YARN 调度器负载模拟器 (SLS)

• YARN 调度器负载模拟器 (SLS)
  • 概述
    • 概述
    • 目标
    • 架构
    • 用例
  • 用法
    • 步骤 1：配置 Hadoop 和模拟器
    • 步骤 2：运行模拟器
  • 指标
    • 实时跟踪
    • 离线分析
  • 合成负载生成器
  • 附录
    • 资源
    • SLS JSON 输入文件格式
    • SYNTH JSON 输入文件格式
    • 模拟器输入拓扑文件格式

$ cd $HADOOP_ROOT/share/hadoop/tools/sls
$ bin/slsrun.sh
  Usage: slsrun.sh <OPTIONS>
             --tracetype=<SYNTH | SLS | RUMEN>
             --tracelocation=<FILE1,FILE2,...>
             (deprecated --input-rumen=<FILE1,FILE2,...>  | --input-sls=<FILE1,FILE2,...>)
             --output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY>
             [--nodes=<SLS_NODES_FILE>]
             [--track-jobs=<JOBID1,JOBID2,...>]
             [--print-simulation]

 <property>
   <name>yarn.resource-types</name>
   <value>resource-type1, resource-type2</value>
 </property>

 <property>
   <name>yarn.sls.nm.resource-type1</name>
   <value>10</value>
 </property>
 <property>
   <name>yarn.sls.nm.resource-type2</name>
   <value>10</value>
 </property>

{
  "job.start.ms" : 0,
  "am.memory-mb": 2048,
  "am.vcores": 2,
  "am.resource-type1": 2,
  "am.resource-type2": 2,
  "job.tasks" : [ {
    "container.duration.ms":  5000
    "container.memory-mb": 1024,
    "container.vcores": 1,
    "container.resource-type1": 1,
    "container.resource-type2": 1
  }
}

{
  "num.nodes": 3,  // total number of nodes in the cluster
  "num.racks": 1   // total number of racks in the cluster, it divides num.nodes into the racks evenly, optional, the default value is 1
}
{
  "am.type" : "mapreduce", // type of AM, optional, the default value is "mapreduce"
  "job.start.ms" : 0,      // job start time
  "job.end.ms" : 95375,    // job finish time, optional, the default value is 0
  "job.queue.name" : "sls_queue_1", // the queue job will be submitted to
  "job.id" : "job_1",      // the job id used to track the job, optional. The default value, an zero-based integer increasing with number of jobs, is used if this is not specified or job.count > 1
  "job.user" : "default",  // user, optional, the default value is "default"
  "job.count" : 1,         // number of jobs, optional, the default value is 1
  "job.tasks" : [ {
    "count": 1,    // number of tasks, optional, the default value is 1
    "container.host" : "/default-rack/node1",  // host the container asks for
    "container.start.ms" : 6664,  // container start time, optional
    "container.end.ms" : 23707,   // container finish time, optional
    "container.duration.ms":  50000, // duration of the container, optional if start and end time is specified
    "container.priority" : 20,    // priority of the container, optional, the default value is 20
    "container.type" : "map"      // type of the container, could be "map" or "reduce", optional, the default value is "map"
  }, {
    "container.host" : "/default-rack/node3",
    "container.start.ms" : 6665,
    "container.end.ms" : 21593,
    "container.priority" : 20,
    "container.type" : "map"
  }, {
    "container.host" : "/default-rack/node2",
    "container.start.ms" : 68770,
    "container.end.ms" : 86613,
    "container.priority" : 20,
    "container.type" : "map"
  } ]
}
{
  "am.type" : "mapreduce",
  "job.start.ms" : 105204,
  "job.end.ms" : 197256,
  "job.queue.name" : "sls_queue_2",
  "job.id" : "job_2",
  "job.user" : "default",
  "job.tasks" : [ {
    "container.host" : "/default-rack/node1",
    "container.start.ms" : 111822,
    "container.end.ms" : 133985,
    "container.priority" : 20,
    "container.type" : "map"
  }, {
    "container.host" : "/default-rack/node2",
    "container.start.ms" : 111788,
    "container.end.ms" : 131377,
    "container.priority" : 20,
    "container.type" : "map"
  } ]
}

{
  "description" : "tiny jobs workload",    //description of the meaning of this collection of workloads
  "num_nodes" : 10,  //total nodes in the simulated cluster
  "nodes_per_rack" : 4, //number of nodes in each simulated rack
  "num_jobs" : 10, // total number of jobs being simulated
  "rand_seed" : 2, //the random seed used for deterministic randomized runs

  // a list of “workloads”, each of which has job classes, and temporal properties
  "workloads" : [
    {
      "workload_name" : "tiny-test", // name of the workload
      "workload_weight": 0.5,  // used for weighted random selection of which workload to sample from
      "queue_name" : "sls_queue_1", //queue the job will be submitted to

    //different classes of jobs for this workload
       "job_classes" : [
        {
          "class_name" : "class_1", //name of the class
          "class_weight" : 1.0, //used for weighted random selection of class within workload

          //nextr group controls average and standard deviation of a LogNormal distribution that
          //determines the number of mappers and reducers for thejob.
          "mtasks_avg" : 5,
          "mtasks_stddev" : 1,
          "rtasks_avg" : 5,
          "rtasks_stddev" : 1,

          //averge and stdev input param of LogNormal distribution controlling job duration
          "dur_avg" : 60,
          "dur_stddev" : 5,

          //averge and stdev input param of LogNormal distribution controlling mappers and reducers durations
          "mtime_avg" : 10,
          "mtime_stddev" : 2,
          "rtime_avg" : 20,
          "rtime_stddev" : 4,

          //averge and stdev input param of LogNormal distribution controlling memory and cores for map and reduce
          "map_max_memory_avg" : 1024,
          "map_max_memory_stddev" : 0.001,
          "reduce_max_memory_avg" : 2048,
          "reduce_max_memory_stddev" : 0.001,
          "map_max_vcores_avg" : 1,
          "map_max_vcores_stddev" : 0.001,
          "reduce_max_vcores_avg" : 2,
          "reduce_max_vcores_stddev" : 0.001,

          //probability of running this job with a reservation
          "chance_of_reservation" : 0.5,
          //input parameters of LogNormal distribution that determines the deadline slack (as a multiplier of job duration)
          "deadline_factor_avg" : 10.0,
          "deadline_factor_stddev" : 0.001,
        }
       ],
    // for each workload determines with what probability each time bucket is picked to choose the job starttime.
    // In the example below the jobs have twice as much chance to start in the first minute than in the second minute
    // of simulation, and then zero chance thereafter.
      "time_distribution" : [
        { "time" : 1, "weight" : 66 },
        { "time" : 60, "weight" : 33 },
        { "time" : 120, "jobs" : 0 }
     ]
    }
 ]
}

{
  "rack" : "default-rack",
  "nodes" : [ {
    "node" : "node1"
  }, {
    "node" : "node2"
  }, {
    "node" : "node3"
  }]
}