网站建设管理中se是什么意思,服装网站建设准备,南京公共工程建设中心网站,旅游网站建设的好处1.安装
安装步骤按照官网安装即可
官网#xff1a;DolphinScheduler | 文档中心 (apache.org)
版本#xff1a;3.1.5
2.踩坑记录
Q1.大文件无法上传
问题描述#xff1a;
在资源中心中上传文件选择完大文件夹之后#xff0c;选择确认之后确认按钮转了几圈圈之后就没…1.安装
安装步骤按照官网安装即可
官网DolphinScheduler | 文档中心 (apache.org)
版本3.1.5
2.踩坑记录
Q1.大文件无法上传
问题描述
在资源中心中上传文件选择完大文件夹之后选择确认之后确认按钮转了几圈圈之后就没反应了对应服务器上使用ds登录的用户资源目录也没有对应的文件。
问题解决
通过排查发现每到15s。网络请求就会断开。
利用浏览器工具找到请求的js文件。找到请求urlbaseURL中有个参数是timeout 为 15e3(15000ms即15s),找到对应ds安装目录下的ui/asserts/ 对应js脚本文件时间改大一点。
Q2.Flink程序无法提交到yarn
问题描述
创建完Flink-Stream 之后命令总提交失败分析日志DS生成的提交命令为
去服务器上执行发现有以下报错主要是因为Flink的总内存默认为1.5G。
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: TaskManager memory configuration failed: Sum of configured Framework Heap Memory (128.000mb (134217728 bytes)), Framework Off-Heap Memory (512.000mb (536870912 bytes)), Task Off-Heap Memory (512.000mb (536870912 bytes)), Managed Memory (1024.000mb (1073741824 bytes)) and Network Memory (158.720mb (166429984 bytes)) exceed configured Total Flink Memory (1.550gb (1664299824 bytes))
解决
对应的参数调整成以下配置项即可应该有其他更好的解法暂时没找到......
对应JobManager内存数 和 taskManager内存数分别用以下两个参数代替原有选项置空。
-Djobmanager.memory.process.size1024mb -Dtaskmanager.memory.process.size6144mb Q3.提交完flinkStream 之后在ResourceManagerWebUi上一直处于create状态
Q3问题描述
使用root用户在对应节点的服务器提交任务任务可以成功被提交并且可以running起来。
但是切换完ds租户提交任务之后也可以成功提交但是任务一直处于created 的状态并且在一段时间之后会失败并报错
org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:91)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)at akka.actor.Actor$class.aroundReceive(Actor.scala:517)at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)at akka.actor.ActorCell.invoke(ActorCell.scala:561)at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)at akka.dispatch.Mailbox.run(Mailbox.scala:225)at akka.dispatch.Mailbox.exec(Mailbox.scala:235)at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeoutat java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)... 31 moreCaused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeoutat org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)... 24 moreCaused by: java.util.concurrent.TimeoutException: Timeout has occurred: 300000 ms... 25 moreException in thread Thread-5 java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration classloader.check-leaked-classloader.at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164)at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:183)at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2647)at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:2905)at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2864)at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2838)at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2715)at org.apache.hadoop.conf.Configuration.get(Configuration.java:1186)at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1774)at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)
主要看 Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
Q3问题解决
找到对应node节点的flink安装目录在对应 flink 启动脚本第一行添加一行代码
export HADOOP_CLASSPATHhadoop classpath其含义为运行shell命令hadoop classpath 并将输出的值赋予给HADOOP_CLASSPATH环境变量。
然后重新提交flink任务即可在ResourceManagerWebUi看到对应DS租户提交的Flink任务。 Q4.关于一些僵尸任务前端不能删除
Q4问题描述
1.对于DS的任务管理中任务定义没有删除的操作。已经删除了的工作流中的任务定义依然存在也不能通过前端删除。 2.在任务示例中实时任务也没有删除选项但会存在僵尸任务示例。
Q4问题解决
对于配置了mysql环境的DS。
1.任务定义表为t_ds_process_instance根据名称字段name去删除对应任务定义即可。
2.任务实例表为t_ds_task_instance根据需要条件删除任务实例即可。
