Changelog 1.5 version
1.5.4
released 30th April 2021
Client
Digest authentition: using the realm of the 401 response to an initial incomplete request for an immediate second complete request including realm.
There are cases where the client of a digest authentication has no knowledge of the realm value. For this scenario, a case in which the realm contained in the 401 response is immediately used should be added.
Core
ktor-html-builder: exceptions in DSL are swallowed when StatusPages feature is installed
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/541
Simple demo:
Expected behavior: I would expect the default to involve a 500 and the exception being logged somewhere, hopefully customizable via StatusPages. Rendering part of the DSL's execution with a 200 is highly misleading, to say the least.
ByteBufferChannel.readRemaining suspend when should not
Reproducer:
suspend fun main() {
val channel = ByteChannel(true)
println(channel.availableForRead) //prints 0
channel.writePacket {
writeText("hello")
}
println(channel.availableForRead) //prints 5
val packet = channel.readRemaining(channel.availableForRead.toLong()) //suspend...
// val packet = channel.readPacket(channel.availableForRead)
println(packet.readText())
}
Channel has 5 bytes available for read.
Contract of availableForRead
says, that:
/**
* Returns number of bytes that can be read without suspension. Read operations do no suspend and return
* immediately when this number is at least the number of bytes requested for read.
*/
public actual val availableForRead: Int
But in readRemaining
it suspends on trying to read...
Using readPacket
it works ok.
Using ByteChannelSequential on JVM don't suspend on both calls. On K/N the same - both don't suspends.
Looks like it regression after KTOR-1268. Caused by this change: https://github.com/ktorio/ktor/pull/2214/files#diff-408be3eaf4a8ec196449c0483c1525a24450590d9473ebcc3a5bc3798326e3e7L2234-R2222
Now channel is trying to read one byte, even if it's not needed.
Reproduced on:
kotlin: 1.4.32
ktor: 1.5.3
K/N: ByteChannelSequential.readPacket freeze
Minimal reproducer:
on K/N:
fun main() = runBlocking {
val channel = ByteChannel(true)
channel.writePacket {
writeText("h")
}
channel.close()
val packet = withTimeoutOrNull(1000) {
// val packet = channel.readRemaining(5L)
channel.readPacket(5)
}
println(packet?.readText())
}
Channel closed, but readPacket freezes forever - even not fail by timeout.
Same behavior is if using ByteChannelSequentialJVM(IoBuffer.Empty, true)
on K/JVM.
If using ByteBufferChannel on JVM - behavior is ok, exception is thrown:
Exception in thread "main" kotlinx.coroutines.channels.ClosedReceiveChannelException: Unexpected EOF: expected 4 more bytes
at io.ktor.utils.io.ByteBufferChannel.readFullySuspend(ByteBufferChannel.kt:598)
at io.ktor.utils.io.ByteBufferChannel.readFully(ByteBufferChannel.kt:590)
at io.ktor.utils.io.ByteBufferChannel.readPacketSuspend(ByteBufferChannel.kt:806)
at io.ktor.utils.io.ByteBufferChannel.readPacket$suspendImpl(ByteBufferChannel.kt:792)
at io.ktor.utils.io.ByteBufferChannel.readPacket(ByteBufferChannel.kt)
at io.ktor.utils.io.ByteReadChannelKt.readPacket(ByteReadChannel.kt:202)
at AppKt$main$1$packet$1.invokeSuspend(App.kt:57)
If to use readRemaining
- for any channel, behavior is the same, it reads packet with one letter and prints it.
Original issue: TCP socket freeze on K/N
Minimal reproducer:
Server (K/JVM):
@OptIn(InternalAPI::class)
suspend fun main() {
val connection = aSocket(SelectorManager()).tcp().bind(port = 8000).accept().connection()
while (true) {
delay(1000)
connection.output.writePacket(buildPacket {
writeText("1".repeat(128))
})
connection.output.flush()
}
}
Client (K/Native, K/JVM - same code):
@OptIn(InternalAPI::class)
fun main() = runBlocking {
val connection = aSocket(SelectorManager()).tcp().connect("0.0.0.0", port = 8000).connection()
while (true) {
val packet = withTimeoutOrNull(5000) {
println("await")
val packet = connection.input.readRemaining(128)
// val packet = connection.input.readPacket(128)
require(packet.remaining == 128L) { "Length should be 128 but was ${packet.remaining}" }
packet
}
println("received: ${packet?.readText()}")
}
}
Here, if to start server, then, start client, on client side there will be output:
await
received: 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
await
received: 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
After several seconds, stop the server.
Client on K/Native fails with:
Uncaught Kotlin exception: kotlin.IllegalArgumentException: Length should be 128 but was 0
It's expected, channel closed, and no bytes to read
Client on K/JVM fails with same exception:
Exception in thread "main" java.lang.IllegalArgumentException: Length should be 128 but was 0
Also expected.
But If to replace readRemaining
with readPacket
and do the same (start server, start client, stop server after several seconds):
Client on K/Native just freezes - prints only await
and don't fail on 2 seconds timeout.
Not expected at all!
Client on K/JVM fail with:
Exception in thread "main" kotlinx.coroutines.channels.ClosedReceiveChannelException: Unexpected EOF: expected 128 more bytes
at io.ktor.utils.io.ByteBufferChannel.readFullySuspend(ByteBufferChannel.kt:598)
at io.ktor.utils.io.ByteBufferChannel$readFullySuspend$1.invokeSuspend(ByteBufferChannel.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:86)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:61)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at AppKt.main(App.kt:24)
at AppKt.main(App.kt)
But this is expected.
BTW, if to reduce timeout to 2 seconds, K/JVM will fail by timeout for readRemaining
case, but K/N will fail with expected exception.
Reproduced on:
kotlin: 1.4.32
ktor: 1.5.3
os: macos
K/N target: macos, both release and debug
Docs
Timeout documentation is inaccurate
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/268
https://ktor.io/clients/http-client/features/timeout.html
The Client Timeout Feature states that "By default all these timeouts are infinite"; this doesn't seem to be the case with either the Apache or CIO engines where unless I explicitly set a timeout I get:
Apache
io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=http://localhost:1080/api/import/tickets/v1/file, socket_timeout=unknown] ms
at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:170)
at io.ktor.client.engine.apache.ApacheHttpRequestKt$sendRequest$$inlined$suspendCancellableCoroutine$lambda$2.failed(ApacheHttpRequest.kt:44)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: 10,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
... 11 common frames omitted
CIO (which doesn't seem to accept specifying timeouts at all)
kotlinx.coroutines.TimeoutCancellationException: Timed out waiting for 15000 ms
at kotlinx.coroutines.TimeoutKt.TimeoutCancellationException(Timeout.kt:149)
at kotlinx.coroutines.TimeoutCoroutine.run(Timeout.kt:119)
at kotlinx.coroutines.EventLoopImplBase$DelayedRunnableTask.run(EventLoop.common.kt:493)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:272)
at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:68)
at java.base/java.lang.Thread.run(Thread.java:834)
Add links to complete examples (github/zip) on every quickstart sub-page
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/96
Pages like https://ktor.io/quickstart/quickstart/gradle.html lack ready-to-download (and use) example projects
Update documentation for FatJar generation in ktor.io
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1004
Ktor Version
1.1.3 with Tomcat and Gradle
Operating System
Operating system: MacOS High Sierra
Feedback
By following the steps in documentation to generate a FatJar, whenever I try to execute my Ktor application by using the command java -jar pathToFatJar
, I receive de following error:
Exception in thread "main" java.lang.UnsupportedOperationException: Packages and file facades are not yet supported in Kotlin reflection. Meanwhile please use Java reflection to inspect this class: class com.tidelevel.ApplicationKt
at kotlin.reflect.jvm.internal.KClassImpl.reportUnresolvedClass(KClassImpl.kt:301)
at kotlin.reflect.jvm.internal.KClassImpl.access$reportUnresolvedClass(KClassImpl.kt:43)
at kotlin.reflect.jvm.internal.KClassImpl$Data$descriptor$2.invoke(KClassImpl.kt:53)
at kotlin.reflect.jvm.internal.KClassImpl$Data$descriptor$2.invoke(KClassImpl.kt:44)
at kotlin.reflect.jvm.internal.ReflectProperties$LazySoftVal.invoke(ReflectProperties.java:92)
at kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:31)
at kotlin.reflect.jvm.internal.KClassImpl$Data.getDescriptor(KClassImpl.kt)
at kotlin.reflect.jvm.internal.KClassImpl$Data$objectInstance$2.invoke(KClassImpl.kt:106)
at kotlin.reflect.jvm.internal.ReflectProperties$LazyVal.invoke(ReflectProperties.java:62)
at kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:31)
at kotlin.reflect.jvm.internal.KClassImpl$Data.getObjectInstance(KClassImpl.kt)
at kotlin.reflect.jvm.internal.KClassImpl.getObjectInstance(KClassImpl.kt:239)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createModuleContainer(ApplicationEngineEnvironmentReloading.kt:328)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:317)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:275)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:127)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:247)
at io.ktor.server.tomcat.TomcatApplicationEngine.start(TomcatApplicationEngine.kt:108)
at io.ktor.server.tomcat.EngineMain.main(EngineMain.kt:16)
This error just happen when trying to execute the FatJar in command line, everything works fine when running in Intellij. I have seen similar issues at:
- https://github.com/ktorio/ktor/issues/386
- https://stackoverflow.com/questions/54282790/ktor-running-fat-jar-throws-java-lang-unsupportedoperationexceptionpackages-an
By changing the mainClassName
property to correspond to my own Application class, everything seems to work fine! I am not sure if the documentation needs to be updated in this page or if I did something wrong.
Hope I receive an answer from you!
Documentation for GraalVM
Documentation on Cancelling Requests in Client.
Should cover multiplatform scenarios also
Requests page has incorrect code example for post query
https://ktor.io/clients/http-client/quick-start/requests.html -- This is another page that describes post requests (among with other requests).
First of all, it does not teach what is contentType
parameter of the TextContent
constructor (please, add explanation)
Second of all, the only example about using JsonFeature
tells:
If you install the JsonFeature, and set the content type to application/json you can use arbitrary instances as the body, and they will be serialized as JSON:
but in the code snippet provided for this example there is no contentType
set, and it has incorrect code (please, fix):
// !!!!!!! @Serializable is missing here !!!!!!
data class HelloWorld(val hello: String)
val client = HttpClient(Apache) {
install(JsonFeature) {
serializer = GsonSerializer {
// Configurable .GsonBuilder
serializeNulls()
disableHtmlEscaping()
}
}
}
client.post<Unit> {
url("http://127.0.0.1:8080/")
body = HelloWorld(hello = "world")
// !!!!!!! contentType(ContentType.Application.Json) is missing here !!!!!!
}
Fat JAR documentation lacks information
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/386
Followed instruction here https://ktor.io/servers/deploy.html for fat jar deployment
When I run java -jar myjar.jar , i get main class could'nt be found in io.ktor.server.netty.DevelopmentEngine.
Project runs fine in IDEA
Any idea why?
Generator
Wizard: Remove kotlinx-html Space Repository from Gradle Template
Including Kotlinx HTML DSL as the templating language results in the following line in the configuration:
maven { url = uri("https://maven.pkg.jetbrains.space/public/p/kotlinx-html/maven") }
Since kotlinx-html-jvm:0.7.3 is already published on mavenCentral, there is no need for this line anymore going forward, and it can be removed. (Relates to KTOR-2203, which can be closed, I suppose.)
No templates/thymeleaf directory in new project
IntelliJ IDEA 2021.1 with Ktor plugin 1.5.3
- New Project - Ktor
- Add Feature - Thymeleaf
- Open Templating.kt file
Actual result: templates/thymeleaf/
is mentioned in code but there is no such directory and index
file in project
fun Application.configureTemplating() {
install(Thymeleaf) {
setTemplateResolver(ClassLoaderTemplateResolver().apply {
prefix = "templates/thymeleaf/"
suffix = ".html"
characterEncoding = "utf-8"
})
}
routing {
get("/html-thymeleaf") {
call.respond(ThymeleafContent("index", mapOf("user" to ThymeleafUser(1, "user1"))))
}
}
}
"IllegalArgumentException: Neither port nor sslPort specified" when running via Gradle Application plugin
To reproduce create a new project via the IDEA plugin without any feature and run generated application using Gradle:
./gradlew run
As a result the app crashes with the following error:
> Task :run FAILED
2021-04-08 16:20:57.086 [main] TRACE Application - No configuration provided: neither application.conf nor system properties nor command line options (-config or -P:ktor...=) provided
Exception in thread "main" java.lang.IllegalArgumentException: Neither port nor sslPort specified. Use command line options -port/-sslPort or configure connectors in application.conf
at io.ktor.server.engine.CommandLineKt$commandLineEnvironment$environment$1.invoke(CommandLine.kt:142)
at io.ktor.server.engine.CommandLineKt$commandLineEnvironment$environment$1.invoke(CommandLine.kt)
at io.ktor.server.engine.ApplicationEngineEnvironmentBuilder.build(ApplicationEngineEnvironment.kt:110)
at io.ktor.server.engine.ApplicationEngineEnvironmentKt.applicationEngineEnvironment(ApplicationEngineEnvironment.kt:46)
at io.ktor.server.engine.CommandLineKt.commandLineEnvironment(CommandLine.kt:61)
at io.ktor.server.netty.EngineMain.main(EngineMain.kt:21)
Wizard: Different serialization engines should be made mutually exclusive
Adding "Gson" and "Kotlinx.Serialization" dependencies in the same wizard project generates the following code:
install(ContentNegotiation) {
gson {
}
}
install(ContentNegotiation) {
json()
}
// . . .
routing {
get("/json/gson") {
call.respond(mapOf("hello" to "world"))
}
}
routing {
get("/json/kotlinx-serialization") {
call.respond(mapOf("hello" to "world"))
}
}
This (expectedly) crashes at first run:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Caused by: io.ktor.application.DuplicateApplicationFeatureException: Conflicting application feature is already installed with the same key as `ContentNegotiation`
at io.ktor.application.ApplicationFeatureKt.install(ApplicationFeature.kt:82)
at com.example.ApplicationKt.module(Application.kt:27)
at com.example.ApplicationKt.module$default(Application.kt:21)
Infrastructure
Curl Add Path to new default Homebrew installation
If you want to build KTOR on a new fresh Mac, you need curl. The primary option to install curl is using Homebrew, which change the default installation path from /usr/local
to /opt/homebrew
from 2.5.10. The installation path will only change for new brew installation, to prevent crashing stable installation. On Apple M1 chip, it will always use the new installation path by default.
Fix:
Add "/opt/homebrew/opt/curl/include/curl",
to the build script
IntelliJ IDEA Plugin
Wrong parameter passed to embeddedServer on generated wizard with CIO option
When selecting the CIO option in the Project Wizard, it uses "Netty" as parameter to the embeddedServer function as opposed to CIO.
Ktor plugin memory leak in New Project wizard
IJ 212 Nightly
- Open New Project wizard
- Select Ktor
- Close New Project wizard
2021-04-30 10:44:27,515 [ 181985] ERROR - tellij.openapi.util.ObjectTree - Memory leak detected: 'io.ktor.initializr.intellij.features.KtorFeaturesStep' of class io.ktor.initializr.intellij.features.KtorFeaturesStep is registered in Disposer but wasn't disposed.
Register it with a proper parentDisposable or ensure that it's always disposed by direct Disposer.dispose call.
See https://jetbrains.org/intellij/sdk/docs/basics/disposers.html for more details.
The corresponding Disposer.register() stacktrace is shown as the cause:
java.lang.RuntimeException: Memory leak detected: 'io.ktor.initializr.intellij.features.KtorFeaturesStep' of class io.ktor.initializr.intellij.features.KtorFeaturesStep is registered in Disposer but wasn't disposed.
Register it with a proper parentDisposable or ensure that it's always disposed by direct Disposer.dispose call.
See https://jetbrains.org/intellij/sdk/docs/basics/disposers.html for more details.
The corresponding Disposer.register() stacktrace is shown as the cause:
at com.intellij.openapi.util.ObjectTree.assertIsEmpty(ObjectTree.java:227)
at com.intellij.openapi.util.Disposer.assertIsEmpty(Disposer.java:167)
at com.intellij.openapi.util.Disposer.assertIsEmpty(Disposer.java:162)
at com.intellij.openapi.application.impl.ApplicationImpl.disposeContainer(ApplicationImpl.java:246)
at com.intellij.openapi.application.impl.ApplicationImpl.disposeSelf(ApplicationImpl.java:263)
at com.intellij.openapi.application.impl.ApplicationImpl.doExit(ApplicationImpl.java:650)
at com.intellij.openapi.application.impl.ApplicationImpl.exit(ApplicationImpl.java:603)
at com.intellij.openapi.application.impl.ApplicationImpl.exit(ApplicationImpl.java:592)
at com.intellij.openapi.application.ex.ApplicationEx.exit(ApplicationEx.java:73)
at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeFrame$3.windowClosing(WelcomeFrame.java:117)
at java.desktop/java.awt.AWTEventMulticaster.windowClosing(AWTEventMulticaster.java:357)
at java.desktop/java.awt.AWTEventMulticaster.windowClosing(AWTEventMulticaster.java:357)
at java.desktop/java.awt.Window.processWindowEvent(Window.java:2090)
at java.desktop/javax.swing.JFrame.processWindowEvent(JFrame.java:298)
at java.desktop/java.awt.Window.processEvent(Window.java:2049)
at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5027)
at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2784)
at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859)
at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751)
at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:896)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:768)
at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:434)
at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:825)
at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:433)
at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:842)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:487)
at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
Caused by: java.lang.Throwable
at com.intellij.openapi.util.ObjectNode.<init>(ObjectNode.java:31)
at com.intellij.openapi.util.ObjectTree.createNodeFor(ObjectTree.java:99)
at com.intellij.openapi.util.ObjectTree.register(ObjectTree.java:60)
at com.intellij.openapi.util.Disposer.register(Disposer.java:73)
at com.intellij.util.Alarm.<init>(Alarm.java:130)
at com.intellij.openapi.ui.LoadingDecorator.<init>(LoadingDecorator.java:54)
at com.intellij.openapi.ui.LoadingDecorator.<init>(LoadingDecorator.java:47)
at com.intellij.openapi.ui.LoadingDecorator.<init>(LoadingDecorator.java:43)
at com.intellij.ui.components.JBLoadingPanel.lambda$new$0(JBLoadingPanel.java:29)
at com.intellij.ui.components.JBLoadingPanel.<init>(JBLoadingPanel.java:38)
at com.intellij.ui.components.JBLoadingPanel.<init>(JBLoadingPanel.java:29)
at io.ktor.initializr.intellij.features.KtorFeaturesStep.<init>(KtorFeaturesStep.kt:33)
at io.ktor.initializr.intellij.KtorModuleBuilder.createWizardSteps(KtorModuleBuilder.kt:63)
at com.intellij.ide.util.newProjectWizard.StepSequence.addStepsForBuilder(StepSequence.java:52)
at com.intellij.ide.projectWizard.ProjectTypeStep.<init>(ProjectTypeStep.java:234)
at com.intellij.ide.projectWizard.NewProjectWizard.init(NewProjectWizard.java:57)
at com.intellij.ide.projectWizard.NewProjectWizard.<init>(NewProjectWizard.java:46)
at com.intellij.ide.actions.NewProjectAction.actionPerformed(NewProjectAction.java:20)
at com.intellij.openapi.actionSystem.ex.ActionUtil.performAction(ActionUtil.java:247)
at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAware(ActionUtil.java:236)
at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAwareWithCallbacks(ActionUtil.java:229)
at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeScreenActionsUtil.performAnActionForComponent(WelcomeScreenActionsUtil.java:96)
at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeScreenActionsUtil$ToolbarTextButtonWrapper$1.actionPerformed(WelcomeScreenActionsUtil.java:56)
at java.desktop/javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967)
at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308)
at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405)
at java.desktop/javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262)
at java.desktop/javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:270)
at java.desktop/java.awt.Component.processMouseEvent(Component.java:6652)
at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3345)
at java.desktop/java.awt.Component.processEvent(Component.java:6417)
at java.desktop/java.awt.Container.processEvent(Container.java:2263)
at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5027)
at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859)
at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918)
at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547)
at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488)
at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307)
at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2784)
at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859)
at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751)
at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:896)
at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.java:828)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:765)
... 11 more
Unable to add Ktor module to KMM project
To reproduce:
- Create KMM project
- Add Ktor module
Expected: Ktor module is added
Actual: Ktor module is not added with an error message "Error adding module to project: Module already exists"
On the attached video root module is called "karta" and new Ktor module is "zzz", but the error eventually says that module "karta" already exists.
Tried to restart IDE and and invalidate caches.
IU-211.6693.111, JRE 11.0.10+9-b1341.35x64 JetBrains s.r.o., OS Mac OS X(x86_64) v11.2.3, screens 5120.0x2880.0, 5120.0x2880.0, 2880.0x5120.0; Retina
Empty get/post routes with non-empty context are not visible in Endpoints VIew
Example:
route("path1/path2") {
get {}
post {}
}
path1/path2
will not be visible in Endpoints View while it must be there twice: for POST and for GET methods.
Server
The route "/" ends up as "///" in Route.toString()
This test passes -- note the ///
@Test
internal fun routeSlashes() {
withTestApplication({
routing {
get("/") {
call.respond("OK: ${(call as RoutingApplicationCall).route}")
}
}
}) {
with(handleRequest(HttpMethod.Get, "/")) {
assertEquals("OK: ///(method:GET)", response.content)
}
}
}
If the route is declared with get("")
, Route.toString()
results in /(method:GET)
, which is what I would expect in both cases.
Other
Subclass IosHttpRequestException from IoException or HttpRequestException
If you have a common module containing the api for your app, you can't annotate the functions with @Throws(IosHttpRequestException::class)
, because IosHttpRequestException
is defined in the iOS client only and is a subclass from Exception
. Adding @Throws(Exception::class)
is not a good choice, because a possible wrong request builder is catched too (eg invalide urlString).
class API(val client: HttpClient) {
@Throws(IosHttpRequestException::class) // not possible in commonMain
suspend fun users() = client.get<List<User>>("/users")
}
solution
class API(val client: HttpClient) {
@Throws(IOException::class)
suspend fun users() = client.get<List<User>>("/users")
}
Fix testRoutingWithTracing on windows
ConcurrentMap remove does not work on elements with the same hashCodes
Fix version publishing for Kotlinx.html
Fix `releaseVersion` property
1.5.3
released 5th April 2021
Client
java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
After updating from ktor 1.4.3 to 1.5.2 and running my integration tests using ktor I'm running into:
java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
at io.ktor.util.Attributes$DefaultImpls.get(Attributes.kt:29)
at io.ktor.util.AttributesJvmBase.get(AttributesJvm.kt:15)
at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:28)
at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
at io.ktor.client.features.HttpCallValidator.validateResponse(HttpCallValidator.kt:55)
at io.ktor.client.features.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:134)
at io.ktor.client.features.HttpCallValidator$Companion$install$3.invoke(HttpCallValidator.kt)
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:98)
at ???(Coroutine boundary.?(?)
at app.becoach.roboter.CoacheeRoboter$login$1.invokeSuspend(CoacheeRoboter.kt:26)
Caused by: java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
at io.ktor.util.Attributes$DefaultImpls.get(Attributes.kt:29)
at io.ktor.util.AttributesJvmBase.get(AttributesJvm.kt:15)
at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:28)
at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
at io.ktor.client.features.HttpCallValidator.validateResponse(HttpCallValidator.kt:55)
at io.ktor.client.features.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:134)
at io.ktor.client.features.HttpCallValidator$Companion$install$3.invoke(HttpCallValidator.kt)
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:98)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Do you happen to have any idea what's causing this? My unit tests are getting 4xx error codes from the backend. I've got a custom Feature installed where upon unauthorized error codes, I'm re-issuing a new token.
My feature looks like this:
internal class AuthFeature(
private val credentials: String,
private val tokenStorage: TokenStorage,
private val logger: Logger
) {
class Config {
lateinit var credentials: String
lateinit var tokenStorage: TokenStorage
lateinit var logger: Logger
}
companion object Feature : HttpClientFeature<Config, AuthFeature> {
override val key: AttributeKey<AuthFeature> = AttributeKey("AuthFeature")
/**
* The Mutex is required in order to only refresh once when multiple requests
* are being triggered simultaneously e.g. at app startup.
*/
private val mutex = Mutex()
override fun prepare(block: Config.() -> Unit): AuthFeature {
val config = Config().apply(block)
return AuthFeature(
credentials = config.credentials,
tokenStorage = config.tokenStorage,
logger = config.logger,
)
}
@KtorExperimentalAPI
override fun install(feature: AuthFeature, scope: HttpClient) {
scope.requestPipeline.intercept(HttpRequestPipeline.State) {
val accessToken = feature.tokenStorage.getAccessToken()
val isRefreshingToken = context.url.buildString().endsWith("oauth/token")
val header = if (accessToken != null && !isRefreshingToken) "Bearer $accessToken" else feature.credentials
context.headers.append(HttpHeaders.Authorization, header)
}
requireNotNull(scope.feature(HttpSend)).intercept { call, _ ->
if (call.response.status == HttpStatusCode.Unauthorized) {
val mutexId = uuid4().toString()
val tag = "${key.name}-$mutexId"
feature.logger.i(tag, "Got ${HttpStatusCode.Unauthorized.value}")
val beforeMutexAccessToken = feature.tokenStorage.getAccessToken()
val request = call.request
mutex.withLock {
val inMutexAccessToken = feature.tokenStorage.getAccessToken()
val needsToRefresh = beforeMutexAccessToken == inMutexAccessToken
if (needsToRefresh) {
feature.tokenStorage.putAccessToken(null) // Invalidate access token.
}
val refreshToken = feature.tokenStorage.getRefreshToken()
if (refreshToken != null) {
if (needsToRefresh) {
feature.logger.i(tag, "Issuing new access token")
val refreshResponse = scope.refreshToken(refreshToken)
feature.logger.i(tag, "Updating access & refresh token")
feature.tokenStorage.putAccessToken(refreshResponse.accessToken)
feature.tokenStorage.putRefreshToken(refreshResponse.refreshToken)
} else {
feature.logger.i(tag, "Other request already refreshed token")
}
// Build previous request once again.
val requestBuilder = HttpRequestBuilder()
requestBuilder.takeFrom(request)
// Force the new token by explicitly setting it.
requestBuilder.headers[HttpHeaders.Authorization] = "Bearer ${feature.tokenStorage.getAccessToken()}"
feature.logger.i(tag, "Retrying previous request with new access token")
execute(requestBuilder).also {
feature.logger.i(tag, "Finished previous request")
}
} else {
feature.logger.i(tag, "No refresh token and hence leaving as is")
call
}
}
} else {
call
}
}
}
}
}
Fix flaky CIOSustainabilityTest.testBlockingConcurrency[jvm]
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:38)
at io.ktor.network.sockets.SocketImpl$connect$1.invokeSuspend(SocketImpl.kt)
at (Coroutine boundary. ( )
at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)
Ktor 1.5.2: Can't resolve 'node-fetch' on libs produced by jsBrowserProductionLibraryDistribution
I'm using Ktor client with an MPP project.
After upgrading to 1.5.2, I started having this error on a yarn
project that is using my js module as a javascript dependency:
./node_modules/mp-game-clib/mp-game-clib.js
Module not found: Can't resolve 'node-fetch' in '/Users/brunomed/git/pok-p2p-game-demo/react-app/node_modules/mp-game-clib'
That's the relevant Gradle config for the library project:
js(IR) {
useCommonJs()
browser {
binaries.library()
webpackTask {
enabled = false
}
testTask {
useKarma {
useChromeHeadless()
webpackConfig.cssSupport.enabled = true
}
}
}
}
...
}
sourceSets {
val commonMain by getting {
dependencies {
implementation(project(":client-server"))
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.1.0")
implementation("io.ktor:ktor-client-core:$ktorVersion")
implementation("io.ktor:ktor-client-websockets:$ktorVersion")
implementation("com.benasher44:uuid:0.2.3")
// https://youtrack.jetbrains.com/issue/KTOR-541
runtimeOnly(npm("text-encoding", "0.7.0"))
runtimeOnly(npm("abort-controller", "3.0.0"))
runtimeOnly(npm("node-fetch", "2.6.1")) <== Had to add this to fix the problem for now, I didn't need it with Ktor 1.5.1
}
}
...
}
Not sure if I'm doing something wrong here, but adding those npm
dependencies is the only way I could find to produce correct package.json
on the jsBrowserProductionLibraryDistribution
goal (previous comment here: https://youtrack.jetbrains.com/issue/KTOR-541#focus=Comments-27-4731438.0-0)
Outdated doc string for FormPart
Documentation for FormPart
says
@param value content, could be [String], [Number] or [Input]
https://github.com/ktorio/ktor/blob/e03bafda3b3d72fcac166e46cf55e5d2d9383660/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/formDsl.kt#L17
But later in this file Input
is deprecated
@Suppress("UNUSED_PARAMETER")
@Deprecated(
"Input is not reusable. Please use [InputProvider] instead.",
level = DeprecationLevel.ERROR,
replaceWith = ReplaceWith("appendInput(key, headers) { /* create fresh input here */ }")
)
public fun append(key: String, value: Input, headers: Headers = Headers.Empty) {
error("Input is not reusable. Please use [InputProvider] instead.")
}
And direct usage will cause exception
public fun formData(vararg values: FormPart<*>): List<PartData> {
...
is Input -> error("Can't use [Input] as part of form: $value. Consider using [InputProvider] instead.")
...
}
Core
I/O readRemaining sometimes looses exception
The function channel.readRemaining
sometimes returns a packet (possibly the empty one) when the channel is closed abnormally.
Upgrade kotlinx.serialization to 1.1.0
Upgrade to coroutines 1.4.3
Docs
Requests topic miss the import statement for call.receiveText
A user gets the unresolved reference "call.receiveText"
error when trying to use examples from https://ktor.io/docs/requests.html
Validate tutorial on Creating HTTP APIs with Ktor
There are reports that the tutorial located at https://play.kotlinlang.org/hands-on/Creating HTTP APIs with Ktor/02_application-init does not work well with the latest version of the Ktor plugin. We recently ported this tutorial over to our docs. Please make sure it is all correct.
Wrong redirect from old documentation page
Right now the page https://ktor.io/clients/http-client/features/auth.html is redirected to https://ktor.io/docs/http-client-features.html but should be redirected to https://ktor.io/docs/features-auth.html instead.
404 page on https://ktor.io/clients/http-client.html#websockets
The first link from the comment https://github.com/ktorio/ktor/issues/507#issuecomment-410173950 gives a 404 page.
Doc Code Snippets contain out of data repositories
Most contain bintray and jcenter repositories. They need to be removed.
Wrong indentation in `Serving Static Content` guide
Some snippets have 4 spaces and some 2
https://ktor.io/docs/serving-static-content.html#serving-individual-files
Generator
Rename build system options in generator
Should be: Gradle Kotlin, Gradle Groovy, Maven
Provide default route in generated application
The wizard generates a new app that compiles and runs, but when clicking on the default route as indicated by the output, it doesn't respond. I think we should provide a default route not only as means of folks knowing that it works, but also as a starting point.
Wizard generates obsolete repositories
Wizard is still referencing Bintray and jcenter which aren't required.
repositories {
mavenLocal()
jcenter()
maven { url = uri("https://kotlin.bintray.com/ktor") }
}
A project generated with Maven using a new plugin contains deprecated API in 'pom.xml'
To reproduce the issue, create a new Ktor project with the Maven build system. The pom.xml
file will contain the following code for the exec-maven-plugin
plugin:
<configuration>
<mainClass>io.ktor.server.netty.DevelopmentEngine</mainClass>
</configuration>
But DevelopmentEngine
is deprecated.
Infrastructure
Autoreload not working with 1.5.x when using embeddedServer NOT in debug mode
The 1.5.x update broke the auto-reload feature.
I made an example repo here, when switching from 1.4.x to 1.5.x the auto-reload break (not working anymore).
You can reproduce by running ./gradlew run
and ./gradlew build -t
and switch versions.
IntelliJ IDEA Plugin
Wizard: Creating Maven project with Kotlinx.Serialization results in Could not find artifact org.jetbrains.kotlin:kotlin-maven-serialization:pom
Could not find artifact org.jetbrains.kotlin:kotlin-maven-serialization:pom:${kotlin.version} in central (https://repo.maven.apache.org/maven2)
I assume this is the offending block:
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-serialization</artifactId>
<version>${kotlin.version}</version>
</dependency>
Link to GitHub for Features is incorrect
If we're linking to the specific code for a feature on GitHub (which should probably be indicated in the metadata for the feature and only linked if present), it should link to the specific feature. Currently all the ones we ship out of the box point to generic https://github.com/ktorio/ktor
Also, please note the text should say
"Source on GitHub" instead of "See features's GitHub"
Wizard: WebSocket feature adds redundant qualifier name
Wizard: "Static Content" feature generates invalid index.html
The feature generates the following code in resources/static/index.html
:
<html>
<head>
</head>
<body>
<h1>Hello Ktor!<h1>
</body>
</html>
As IntelliJ IDEA notes correctly, the h1
tag is not being closed properly. Should be </h1>
.
Remove README From generated projects
With a barebones Ktor project as generated by the Wizard, the stubbed-out code snippets in the README.md cause IntelliJ to believe we're committing very faulty software when importing our project into Git (and, likely, other VCS). IJ shows 50+ errors (not warnings).
This looks very scary, and makes it feel like something is broken with the setup even when there isn't. Please either annotate the README with some kind of flag to let IntelliJ know not to check all the code fences, or otherwise exclude the README file from code analysis.
Tested on:
IntelliJ IDEA 2021.1 Beta (Ultimate Edition)
Build #IU-211.6305.21, built on March 4, 2021
IntelliJ IDEA EAP User
Expiration date: April 4, 2021
Runtime version: 11.0.10+9-b1341.18 x86_64
VM: Dynamic Code Evolution 64-Bit Server VM by JetBrains s.r.o.
macOS 10.14.6
GC: G1 Young Generation, G1 Old Generation
Memory: 4096M
Cores: 8
Registry: ide.images.show.chessboard=true
Non-Bundled Plugins: com.jetbrains.darkPurpleTheme (1.2), org.nik.presentation-assistant (1.0.9), wu.seal.tool.jsontokotlin (3.6.1), org.asciidoctor.intellij.asciidoc (0.32.26), stardust (1.5.2105), com.jetbrains.intellij.api.watcher (6.46.0), intellij.ktor (1.5.1-eap-1)
Kotlin: 211-1.4.21-release-IJ6305.1
Wizard: Gradle Groovy project with Kotlinx Serialization uses inconsistent quotes
plugins {
id 'application'
id 'org.jetbrains.kotlin.jvm' version '1.4.10'
id "org.jetbrains.kotlin.plugin.serialization" version "1.4.10" // <== these should probably also be single quotes – no need to switch.
}
Wizard: Unused import directive in build.gradle.kts
import org.jetbrains.kotlin.gradle.dsl.Coroutines
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
Wizard: Basic run configuration is not always created
If I create a project with ContentNegotiation, Gson, and Routing, I don't get a run configuration automatically configured. I need to press the green gutter play icon, or run the run
Gradle task.
Adding a bunch more features seems to sometimes include a default "ApplicationKt" run configuration.
Wizard: Selecting MAVEN adds multiple Bintray references to pom.xml
For some reason it also add numerous entries
<repositories>
<repository>
<id>jcenter</id>
<url>https://jcenter.bintray.com</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>ktor</id>
<url>https://kotlin.bintray.com/ktor</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>jcenter</id>
<url>https://jcenter.bintray.com</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>ktor</id>
<url>https://kotlin.bintray.com/ktor</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>jcenter</id>
<url>https://jcenter.bintray.com</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>ktor</id>
<url>https://kotlin.bintray.com/ktor</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>jcenter</id>
<url>https://jcenter.bintray.com</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>ktor</id>
<url>https://kotlin.bintray.com/ktor</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>jcenter</id>
<url>https://jcenter.bintray.com</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>ktor</id>
<url>https://kotlin.bintray.com/ktor</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
Reference-style documentation links are not rendered in a plugin
- Create a new Ktor project.
- Go to the Features page and find the Micrometer Metrics feature.
- Find the
[monitoring the JVM][micrometer_jvm_metrics]
text => should be a link
Documentation links don't work in a plugin
- Create a new Ktor project.
- Go to the Features page and find the Micrometer Metrics feature.
- Click any link => doesn't work
New Ktor plugin saves scroll position in feature description
When I scroll feature documentation to some point and then click to another feature the scroll position remains the same. This requires me to scroll to the top to start reading another feature's documentation.
Application structure browsing, routing navigator and edit for Ktor plugin
As a Ktor backend developer I want to have an IDE interface with endpoints list, generated from my routing configuration:
Possible features of endpoints map:
- go to the related code;
- generate a new route wizard;
- open in browser;
- generate dummy request with applicable params (according to data class) for IDEA http-client;
- generate test from request/response;
- show pipelines and interceptors chain;
- show execution timings
Server
Simple API for writing features
Main goals
- Make API easy to understand for users:
a) Get rid of concept of pipelines and phases
b) Make API compact (reduce number of options available, i.e. consider smaller number of "phases")
c) Rework naming to make it intuitive
d) Think about defaults and yet support common case - API must be compatible with old existing features
- All usages of old features rewritten with new API must remain unchanged yet working. This means all
import {...}
blocks should still work for features rewritten to the new API as well as with the old features. - Choosing "phase" must become intuitive and easy
- Old API functionality should remain available for experienced users.
- API should fit into Kotlin best patterns and concepts (DSL is a good option here)
Onboarding (first steps)
Old API
Let's take a look at some small feature that was defined with the old Features API. This example was taken from our official documentation:
class CustomFeature(configuration: Configuration) {
val prop = configuration.prop // Copies a snapshot of the mutable config into an immutable property.
class Configuration {
var prop = "value" // Mutable property.
}
// Implements ApplicationFeature as a companion object.
companion object Feature : ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature> {
// Creates a unique key for the feature.
override val key = AttributeKey<CustomFeature>("CustomFeature")
// Code to execute when installing the feature.
override fun install(pipeline: ApplicationCallPipeline, configure: Configuration.() -> Unit): CustomFeature {
// It is responsibility of the install code to call the `configure` method with the mutable configuration.
val configuration = CustomFeature.Configuration().apply(configure)
// Create the feature, providing the mutable configuration so the feature reads it keeping an immutable copy of the properties.
val feature = CustomFeature(configuration)
// Intercept a pipeline.
pipeline.intercept(…) {
// Perform things in that interception point.
}
return feature
}
}
}
Old Problems:
- This code does not look welcoming to newcomers at all.
- It is not easy to understand.
- It contains a lot of concepts to deal with that are not hidden by the API:
ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature>
,AttributeKey<CustomFeature>
,ApplicationCallPipeline
,pipeline.intercept
- It is not clear where is the feature itself and where is it's configuration: there is a
CustomFeature
that has some properties, and there is also aConfiguration
class that defines configuration, and there is also aFeature
object. It is not 100% obvious how each of them is used and such model confuses unexperienced readers because there is no clear separation between these concepts.
Proposed solution:
- Instead of creating 2 nested classes and an object, extending some weird class
ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature>
now it is sufficient to just create an instance ofKtorFeature
usingcreateFeature { ... }
DSL method, and to define a configuration class for it separately:
class Configuration {
// Properties and configuration functions go here:
...
}
val CustomFeature = createFeature(name = "CustomFeature", createConfiguration = ::Configuration) {
// What feature does goes here:
...
}
This feature can now be used as it was before, with just calling install and configuring it in your main application code:
install(CustomFeature) {
// set mutable config properties
}
Benefits
- Now, users will see much more clear separation between the data and actions, and they will not have to deal with confusing methods and base classes to extend.
- It is fully compatible with the old code that uses features
Extending Ktor HTTP behaviour
Old API
In the old API users had to deal with a concept of phases and pipelines. Below there is a list of them:
ApplicationCallPipeline
Setup
: Phase for preparing call and it's attributes for processingMonitoring
: Phase for preparing call and it's attributes for processingFeatures
: Phase for features. Most features should intercept this phase.Call
: Phase for processing a call and sending a responseFallback
: Phase for handling unprocessed calls
ApplicationSendPipeline
Before
: The earliest phase that happens before any otherTransform
: Transformation phase that can proceed with any supported data like StringRender
: Phase to render any current pipeline subject into [io.ktor.http.content.OutgoingContent], beyond this phase only [io.ktor.http.content.OutgoingContent] should be produced by any interceptorContentEncoding
: Phase for processing Content-Encoding, like compression and partial contentTransferEncoding
: Phase for handling Transfer-Encoding, like if chunked encoding is being done manually and not by engineAfter
: The latest application phase that happens right before engine will send the responseEngine
: Phase for Engine to send the response out to client.
ApplicationReceivePipeline
Before
: Executes before any transformations are madeTransform
: Executes transformationsAfter
: Executes after all transformations
Note, that in the list above phases are ordered!
Also, each feature had an ability to create it's custom phase in any of the 3 pipelines, and insert this phase anywhere in the order of existing phases. In order to use some phase ("intercept a pipeline in a specific phase") a feature was meant to call pipeline.intercept
method on a specific phase.
Here is an example of usage of the old API taken from the MicrometerMetrics
implementation:
override fun install(pipeline: Application, configure: Configuration.() -> Unit): MicrometerMetrics {
...
val phase = PipelinePhase("MicrometerMetrics")
pipeline.insertPhaseBefore(ApplicationCallPipeline.Monitoring, phase)
pipeline.intercept(phase) {
// some actions go here
}
val postSendPhase = PipelinePhase("MicrometerMetricsPostSend")
pipeline.sendPipeline.insertPhaseAfter(ApplicationSendPipeline.After, postSendPhase)
pipeline.sendPipeline.intercept(ApplicationSendPipeline.After) {
// some actions go here
}
Old problems:
- There too many phases and pipelines (3 pipelines with 15 phases total)
- Not obvious which phase to select,
- No defaults for some simple usage (onboarding)
- With user-defined phases and insertBefore/insertAfter it builds a complex mental model of the global order of phases and requires a deep knowledge about phases each feature uses in order to think about the order of execution.
- Concept of phases and pipelines is not intuitive and does not mentally map to the concept of Routing in Ktor. It is not obvious when each phase is executed and what are the pipelines.
Proposed solution:
First of all, there should be simple defaults that would extend basic functionalities that users see in Ktor.
There will be 3 "default" methods that are mapped by name with what call inside a routing block each of them extend:
onCall
: extends what happens when a server is processing any HTTP method callonReceive
: extends behaviour of the server when user writescall.receive()
in aroute { ... }
blockonRespond
: extends behaviour of the server when user writescall.respond()
(call.respondText()
, etc.) in aroute { ... }
block
In code they should be placed inside createFeature
block:
val F = createFeature(...) {
onCall {
...
}
onReceive {
...
}
onRespond {
...
}
}
Such defaults are useful for newbies because they are meant to provide the most commonly used functionality without a requirement to know a lot about other options.
Also, for some more advanced use cases there will be more advanced builders with more options to extend:
-
extendCallHandling { ... }
-- a block that configures what happens when HTTP method call handlingmonitoring { ... }
-- defines actions to perform before the call was processed by any feature (including Routing). It is useful for monitoring and logging (see CallLogging feature) to be executed before any "real stuff" was performed with the call because other features can change it's content as well as add more resource usage etc. while for logging and monitoring it is important to observe the pure (untouched) data.onCall { ... }
-- same asonCall
outside ofextendCallHandling
. Most used extension. It defines what to do with the call that is being currently processed by a server.fallback
-- defines what to do with the call in case request was not handled for some reason. Usually, it is handy to usefallback { call -> ...}
to set the response with some message or status code, or to throw an exception.
-
extendReceiveHandling
supports following callbacks:beforeReceive
— defines actions to perform before any transformations were made to the received content. It is useful for caching.onReceive
— defines how to transform data received from a client.
-
extendResponseHandling
supports following callbacks:beforeResponse
— allows to use the direct result of call processing (seeonCall
)onResponse
— does transformations of the data. Example: you can write a custom serializer wit h it.afterResponse
— allows to calculate some stats on the data that was sent, or to handle errors. It is used in Metrics, CachingHeaders, StatusPages.
Example:
val F = createFeature(...) {
onCall {
// prepare some data
}
extendResponseHandling {
beforeResponse {
// use this data
}
onResponse {
// transform data to a string (i.e. serialize data)
}
}
}
Benefits:
- Now we provide fast-to-access defaults that newbies can use (now it is obvious what to select from: not sure -- use a default)
- There are much less options to select from, and these options are mapped to what users do when they define handling of routes
- Api doesn't introduce any new complex or uncommon concepts (like phases or pipelines, or interceptors)
Managing order of execution between features
Old API
In the old API there was a concept of phases that one could introduce and then insert anywhere with insertPhaseBefore
/ insertPhaseAfter
, and then intercept these phases. One of the most common things it was used for was creating order dependencies between features. This idea is explained here: https://www.ximedes.com/2020-09-17/role-based-authorization-in-ktor/. This was useful to either prepare some data that can be used later by another feature, or to use some data already produced by some other feature.
Old Problems:
- It required using 3 concepts to do 1 thing: use "phase", "insertPhase" and "intercept" in order to "execute one feature before/after another"
- It required explicitly knowing which phases another feature was using.
Proposed solution:
Each feature now internally remembers which phases in which pipelines it intercepts (of course, without revealing this complex and confusing information to users). And createFeature
block also provides following functions inside:
beforeFeature(other) { ... }
afterFeature(other) { ... }
Example:
val F = createFeature(...) {
...
}
val G = createFeature(...) {
beforeFeature(F) {
onCall {
// prepare some data for feature F
}
}
}
Benefits:
Now the API just does what it says: it is simply executing one feature before/after another without introducing any complex entities and without a requirement to manually manage the internals.
Manually managing features execution:
Old API
In the old API it was possible to manually manage execution of the current pipeline. Inside any interceptor it was possible to call the following functions of PipelineContext<, >
:
public interface PipelineContext<TSubject : Any, TContext : Any> : CoroutineScope {
/**
* Object representing context in which pipeline executes
*/
public val context: TContext
/**
* Subject of this pipeline execution that goes along the pipeline
*/
public val subject: TSubject
/**
* Finishes current pipeline execution
*/
public fun finish()
/**
* Continues execution of the pipeline with the given subject
*/
public suspend fun proceedWith(subject: TSubject): TSubject
/**
* Continues execution of the pipeline with the same subject
*/
public suspend fun proceed(): TSubject
}
Also, there were some extension methods over this class:
application
: Current application for the contextcall
: Current call for the contexttransformDefaultContent
: Default outgoing content transformation
Although it introduced really complex concept of subject
to understand, it can be required for some advanced users and we can't get rid of it completely. One of the most popular patterns that use PipelineContext
is the following (taken from VelocityFeature
):
pipeline.sendPipeline.intercept(ApplicationSendPipeline.Transform) { value ->
if (value is VelocityContent) {
val response = feature.process(value)
proceedWith(response)
}
}
It actually allows to write a custom data serializer for a specific type (VelocityContent
in this example). proceedWith(outgoingContent)
allows to continue the execution of our HTTP pipeline with the data serialized to a resulting outgoingContent
.
Old problems
- Really complex and hard to understand but yet useful
- Some members/extensions are not that popular or not required for the API (
application
is not commonly used,context
makes no sense to newbies) proceedWith
andsuject
are useful but are mostly used for serialization features- serialization use case requires knowing about
OutgoingContent
type that is not mapped to any general knowledge, and in fact there is always a default serializer fromString
,ByteChanel
andByteArray
types to theOutgoingContent
.
Proposed solution:
- For the case of serialization we will introduce following simple API that users may default to:
public fun <T : Any> KtorFeature<T>.serializeDataToString(callback: suspend SendExecution.(Any) -> String?): Unit =
onSend { callback(subject)?.let { proceedWith(it) } }
public fun <T : Any> KtorFeature<T>.serializeDataToBytes(callback: suspend SendExecution.(Any) -> ByteArray?): Unit =
onSend { callback(subject)?.let { proceedWith(it) } }
public fun <T : Any> KtorFeature<T>.serializeDataToChanel(callback: suspend SendExecution.(Any) -> ByteWriteChannel?): Unit =
onSend { callback(subject)?.let { proceedWith(it) } }
This API can be used as following:
private class Formula(val x: String, val execute: (String) -> String)
private val F = createFeature("F", {}) {
serializeDataToString { f ->
// return null for irrelevant data types
if (f !is Formula) return@serializeDataToString null
// Do a transformation for the current data type
f.execute(f.x)
}
}
private fun Application.f() {
routing {
get("/") {
call.respond(Formula("cake") { x -> "x * x + 1".replace("x", x) })
}
}
}
Also, for any more complex use cases we will introduce the following wrapper over PipelineContext<SubjectT, ApplicationCall>
. It is meant to hide unneeded methods and extensions without limiting functionality too much:
public inline class Execution<SubjectT : Any>(private val context: PipelineContext<SubjectT, ApplicationCall>) {
public suspend fun proceed(): SubjectT = context.proceed()
public suspend fun proceedWith(subectT: SubjectT): SubjectT = context.proceedWith(subectT)
public fun finish(): Unit = context.finish()
public val subject: SubjectT get() = context.subject
public val call: ApplicationCall get() = context.call
...
update cc @leonid.stashevsky Also following methods can be added to handle application-level properties from a feature:
// Useful methods
public val environment: ApplicationEnvironment get() = context.application.environment
public val configuration: ApplicationConfig get() = environment.config
public val port: Int get() = configuration.propertyOrNull("ktor.deployment.port")?.getString()?.toInt() ?: 8080
public val host: String get() = configuration.propertyOrNull("ktor.deployment.host")?.getString() ?: "0.0.0.0"
// Following calback can be used to close resources that a feature has allocated (if needed):
public fun onShutdown(callback: suspend () -> Unit) {
GlobalScope.launch(context.coroutineContext) {
callback()
}
}
}
Also there are type aliases for executions of the different stages:
public typealias CallExecution = Execution<Unit>
public typealias ReceiveExecution = Execution<ApplicationReceiveRequest>
public typealias SendExecution = Execution<Any>
Each of the methods has a corresponding Execution
type as a reciever for a callback, for example:
public override fun onCall(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit { ... }
This means, that inside a feature definition advanced users who know what they do may use proceed
, finish
etc. methods anywhere in the code:
val F = createFeature(...) {
extendRespondHandling {
beforeRespond {
...
if (someCondition)
finish()
...
}
}
}
update cc @leonid.stashevsky Another option for an API
There are following minor issues with the proposed API:
- Naming is quite long for advanced options (
extendCallHandling
as opposed toonCall
), although it is made so intentionally to hide advanced settings from newbies by a longer name. - It sometimes requires writing
this
in Intellij in order to call a completion to learn what's next.
Benefits
- Now it is easy to write serializers
- Now advanced users can still use what they want
- No complex classes are in your face when you write simple code, so nothing will confuse newbies.
Alternative
We can introduce following interfaces (usage will be described later):
public interface OnCall {
public operator fun invoke(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
public fun monitoring(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
public fun fallback(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
}
public interface OnReceive {
public operator fun invoke(callback: suspend ReceiveExecution.(ApplicationCall) -> Unit): Unit
public fun before(callback: suspend ReceiveExecution.(ApplicationCall) -> Unit): Unit
}
public interface OnResponse {
public operator fun invoke(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
public fun before(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
public fun after(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
}
And change FeatureContext
(a supertype of KtorFeature
) to the following:
public interface FeatureContext {
public val onCall: OnCall
public val onReceive: OnReceive
public val onResponse: OnResponse
}
After implementing FeatureContext
it will allow us to write the code in the next way:
val F = makeFeature("F", {}) {
onCall { ... }
onCall.fallback { ... }
onCall.monitoring { ...}
beforeFeature(G) {
onCall { ... }
onReceive.before { ... }
onResponse.after { ... }
}
...
So, in general, following API transformations have been applied:
onCall { ... }
->onCall { ... }
extendCallExecution { fallback { ... } }
->onCall.fallback { ... }
extendCallExecution { monitoring { ... } }
->onCall.monitoring { ... }
onRespond { ... }
->onRespond { ... }
extendRespondExecution { before { ... } }
->onRespond.before { ... }
extendRespondExecution { after { ... } }
->onRespond.after { ... }
onReceive { ... }
->onReceive { ... }
extendReceiveExecution { before { ... } }
->onReceive.before { ... }
Benefits
- Much shorter and readable names
- Advanced functionality is still hidden (now it is hidden by the dot
.
) - Completion perfectly works when one types a dot after an object name to see the options
- Simple things are kept the same because
onCall
,onReceive
andonSend
are executable objects now
Disadvantages
- Not conventional way of writing API, may be uncommon to write dots after kinda-function names
update cc @leonid.stashevsky Compatibility with features written in old API
New API provides an ability for features to interact/depend on each other with beforeFeature
and afterFeature
.
Inside it automatically creates everything that is needed to guarantee the order between features.
But this change requires old features to imlement new interface InterceptionsHolder
in order to make it working with the old features. We can easily do that for builtin features:
/**
* Compatibility class. Interception class for Call phase
* */
public typealias CallInterception = Interception<Unit>
/**
* Compatibility class. Interception class for Receive phase
* */
public typealias ReceiveInterception = Interception<ApplicationReceiveRequest>
/**
* Compatibility class. Interception class for Send phase
* */
public typealias ResponseInterception = Interception<Any>
/**
* Compatibility class. Every feature that needs to be compatible with `beforeFeature(...)` and `afterFeature(...)`
* needs to implement this class. It defines a list of interceptions (see [Interception]) for phases in different pipelines
* that are being intercepted by the current feature.
*
* It is needed in order to get first/last phase in each pipeline for the current feature and create a new phase
* that is strictly earlier/later in this pipeline than any interception of the current feature.
*
* Note: any [KtorFeature] instance automatically fills these fields and there is no need to implement them by hand.
* But if you want to use old feature API based on pipelines and phases and to make it available for `beforeFeature`/
* `afterFeature` methods of the new features API, please consider filling the phases your feature defines by implementing
* this interface [InterceptionsHolder].
* */
public interface InterceptionsHolder {
public val name: String get() = this.javaClass.simpleName
public val fallbackInterceptions: MutableList<CallInterception>
public val callInterceptions: MutableList<CallInterception>
public val monitoringInterceptions: MutableList<CallInterception>
public val beforeReceiveInterceptions: MutableList<ReceiveInterception>
public val onReceiveInterceptions: MutableList<ReceiveInterception>
public val beforeResponseInterceptions: MutableList<ResponseInterception>
public val onResponseInterceptions: MutableList<ResponseInterception>
public val afterResponseInterceptions: MutableList<ResponseInterception>
public fun newPhase(): PipelinePhase = PipelinePhase("${name}Phase${Random.nextInt()}")
}
Question: should we make it internal and hide
pros:
- It reveals old API abstactions and is really complex
- It has a lot of fields to implement and is confusing
- We can implement it in our builtin features and this will cover almost all the cases of the features interaction. It is very rare when someone wants to insert their feature before/after some other third-party feature while it is much more common to insert one relatively to some builtin feature. For the rest of the cases, we can suggest using new API that is easier and does all the work automatically.
cons:
- (Didn't actually find any use-case) probably we'll force old users to rewrite their features with the new API to make them available to the new features.
CORS
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1656
Ktor Version and Engine Used (client or server and name)
1.3.0
Describe the bug
CORS can't pass on some none standard orgin like chrome-extension://mfgdmpfihlmdekaclngibpjhdebndhdj
To Reproduce
open some new page in chrome or other browser,just call any api,CORS throw would error(brower side).
Expected behavior
should be AnyOrigin
pass by CORS, when we config it as host("*")
or anyHost()
Can't override Kotlin module configuration using jackson dsl function
Hi team,
I'm using as Jackson content negotiation feature and can't override Kotlin module configuration when installing using jackson
function.
Here's my code:
install(ContentNegotiation) {
jackson {
registerModule(kotlinModule { nullIsSameAsDefault(true) }) // doesn't work ❌
registerModule(KotlinModule(nullIsSameAsDefault = true)) // also doesn't work ❌
}
}
When I trace to ktor-jackson source code, I found this code:
fun jacksonObjectMapper(): ObjectMapper = jsonMapper { addModule(kotlinModule()) }
Look like Kotlin module already added and there is no way to reconfigure it (?). I can't get any information in issues and documentation page.
My temporary workaround,
install(ContentNegotiation) {
val mapper = jsonMapper { addModule(KotlinModule(nullIsSameAsDefault = true)) } // work great ✔
val jackson = JacksonConverter(mapper)
register(ContentType.Application.Json, jackson)
}
Thank you!
Unexpected exception when using Session feature: "Using blocking primitives on this dispatcher is not allowed"
kotlin: 1.4.20
kotlinx.coroutines: 1.4.2
Sessions feature is configured in a rather standard manner:
install(Sessions) {
cookie<UserIdPrincipal>(
Cookies.SESSION_ID,
storage = directorySessionStorage(sessionStorageRootDir)
) {
cookie.path = "/"
cookie.httpOnly = true
cookie.extensions["SameSite"] = "Lax"
}
}
When it is called during a pipeline execution, an unexpected exception is thrown.
java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
(Coroutine boundary) ~[?:?]
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:259) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) ~[ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) ~[ktor-server-netty-jvm-1.4.3.jar:1.4.3]
Caused by: java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.4.20.jar:1.4.20-release-308 (1.4.20)]
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:342) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:27) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:109) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.async(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.async$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:37) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:31) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_251]
at io.ktor.sessions.BaseCache.getOrCompute(Cache.kt:36) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache.getOrCompute(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseTimeoutCache.getOrCompute(Cache.kt:163) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:63) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.launch(Unknown Source) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:43) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:34) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
NettyApplicationEngine: providing a configureBootstrap in the configuration throws IllegalStateException: group set already
Code snippet:
val server = embeddedServer(Netty, env) {
configureBootstrap = {
group(NioEventLoopGroup(16, Executors.newCachedThreadPool()))
}
}
During NettyApplicationEngine initialization we have:
private val bootstraps = environment.connectors.map { connector ->
ServerBootstrap().apply {
configuration.configureBootstrap(this)
group(connectionEventGroup, workerEventGroup)
The call to group(...) tries to initialize the parent and child EventLoopGroup
s for a second time, which throws the exception above.
Other
InsufficientSpaceException trying to build ByteReadPacket
Periodically (not every time) during running tests in https://github.com/rsocket/rsocket-kotlin using ktor-io exception appeared trying to build ByteReadPacket using buildPacket {}
Exception in thread "DefaultDispatcher-worker-9 @coroutine#122840" io.ktor.utils.io.core.InsufficientSpaceException: Not enough free space to write long integer of 8 bytes, available 7 bytes.
at io.ktor.utils.io.core.BufferPrimitivesKt.writeLong(BufferPrimitives.kt:1081)
at io.ktor.utils.io.core.OutputPrimitivesKt.writeLongFallback(OutputPrimitives.kt:50)
at io.ktor.utils.io.core.OutputPrimitivesKt.writeLong(OutputPrimitives.kt:45)
at io.rsocket.kotlin.frame.KeepAliveFrame.writeSelf(KeepAliveFrame.kt:33)
at io.rsocket.kotlin.frame.Frame.toPacket(Frame.kt:36)
at io.rsocket.kotlin.internal.RSocketState$start$3.invokeSuspend(RSocketState.kt:143)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.selects.SelectBuilderImpl.resumeWith(Select.kt:306)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Invalid assertion for existence of the key in the key store
CommandLine.kt
requireNotNull(getKey(sslKeyAlias, sslPrivateKeyPassword.toCharArray()) == null)
is always successful
Incorrect grammar in exception messages
Multiple instance of "has been" in exception messages such as
Request timeout has been expired
Fix flaky JavaEngineTests.testThreadLeak[jvm]
Fix flaky ExceptionsJvmTest.testConnectionClosedDuringRequest[jvm]
Fix flaky JettyStressTest.highLoadStressTest
HttpClient has strange behaviour when made many parellels requests (JsonFeature with GsonSerializer enabled)
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/821
Hi Ktor team!
I have some problems with the client when make many parallels http requests.
I hope logs and code examples will help you.
Ktor Version
1.0.1
Ktor Engine Used
Android Engine
Features enabled
JsonFeature with GsonSerializer
Platform and hardware
Android 7, 4-core ARM CPU
Feedback
fun test() {
for (i in 1..100) {
println("Start launch")
GlobalScope.launch(Dispatchers.Default) {
val data = client.get<MyResponse>(url)
println("Response completed: $i")
}
println("Finish launch")
}
}`
Logs (please see time, it important, in really one request take about 1-3 seconds):
`
2018-12-24 01:07:47.388 I/System.out: Start launch
2018-12-24 01:07:47.403 I/System.out: Finish launch
// skiped logs: started 100 coroutines
2018-12-24 01:07:47.487 I/System.out: Start launch
2018-12-24 01:07:47.488 I/System.out: Finish launch // last coroutine started
// there is a big pause: in really one request take about 1-3 seconds
2018-12-24 01:07:57.671 I/System.out: Response completed: 9 // first response
// there is a big pause: in really request take about 1-3 seconds
2018-12-24 01:08:27.280 I/System.out: Response completed: 94 // second response
2018-12-24 01:08:28.265 I/System.out: Response completed: 1
2018-12-24 01:08:28.275 I/System.out: Response completed: 2
2018-12-24 01:08:28.285 I/System.out: Response completed: 4
2018-12-24 01:08:28.310 I/System.out: Response completed: 8
2018-12-24 01:08:28.396 I/System.out: Response completed: 80
// skiped logs: responses are distributed evenly in time
2018-12-24 01:08:29.268 I/System.out: Response completed: 92
2018-12-24 01:08:29.280 I/System.out: Response completed: 97
2018-12-24 01:08:29.296 I/System.out: Response completed: 99
2018-12-24 01:08:29.371 I/System.out: Response completed: 100 // last response`
I overrided GsonSerializer and added logs into problematic code.
override suspend fun read(type: TypeInfo, response: HttpResponse): Any {
println("Start response.readText on ${Thread.currentThread().name}")
val text = response.readText()
println("End response.readText on ${Thread.currentThread().name}")
return backend.fromJson(text, type.reifiedType) ?: Any()
}
Logs:
`2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-3 // first start readText
2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-2
2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-1
2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-4
2018-12-24 01:32:40.160 I/System.out: Start response.readText on DefaultDispatcher-worker-1
// skiped 45 logs Start response.readText on ..
2018-12-24 01:33:11.137 I/System.out: End response.readText on DefaultDispatcher-worker-3 // first end readText
2018-12-24 01:33:11.311 I/System.out: Start response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:12.202 I/System.out: Start response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:12.298 I/System.out: Start response.readText on DefaultDispatcher-worker-1
2018-12-24 01:33:12.663 I/System.out: Start response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:12.668 I/System.out: Start response.readText on DefaultDispatcher-worker-1
2018-12-24 01:33:14.052 I/System.out: Start response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:14.091 I/System.out: Start response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:14.238 I/System.out: Start response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:14.313 I/System.out: Start response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:15.475 I/System.out: Start response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:15.486 I/System.out: Start response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:15.576 I/System.out: Start response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:15.646 I/System.out: Start response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:15.746 I/System.out: End response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:15.754 I/System.out: End response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:15.766 I/System.out: End response.readText on DefaultDispatcher-worker-1
2018-12-24 01:33:15.877 I/System.out: End response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:15.914 I/System.out: End response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:15.930 I/System.out: End response.readText on DefaultDispatcher-worker-1
2018-12-24 01:33:15.938 I/System.out: End response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:15.995 I/System.out: End response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:16.051 I/System.out: End response.readText on DefaultDispatcher-worker-1
2018-12-24 01:33:16.101 I/System.out: End response.readText on DefaultDispatcher-worker-2
// skiped many End response.readText on ..
2018-12-24 01:33:16.963 I/System.out: End response.readText on DefaultDispatcher-worker-4
2018-12-24 01:33:16.963 I/System.out: End response.readText on DefaultDispatcher-worker-3
2018-12-24 01:33:16.974 I/System.out: End response.readText on DefaultDispatcher-worker-1
2018-12-24 01:33:16.982 I/System.out: End response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:17.115 I/System.out: Start response.readText on DefaultDispatcher-worker-2
2018-12-24 01:33:17.129 I/System.out: End response.readText on DefaultDispatcher-worker-3 // last end readText`
I think problem in content.readRemaining
method.
Maybe related https://github.com/ktorio/ktor/issues/768, https://github.com/ktorio/ktor/issues/770
I am ready to provide more details let me know.
Thanks!
Reload does not work with "user-local default: JAR manifest" command line shortener
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1266
Ktor Version and Engine Used (client or server and name)
Ktor 1.2.3 embedded Jetty server.
Describe the bug
If server is started with "user-local default: JAR manifest" command line shortener it seems allUrls
in ApplicationEngineEnvironmentReloading.kt
only has a couple of java jars thus it prints:
2019-08-07 14:11:13,088 INFO --- [main] Application : No ktor.deployment.watch patterns match classpath entries, automatic reload is not active
To Reproduce
Error is shown on startup with the mentioned shorten command line option.
Expected behavior
Reloading works. It works for us if we use the classpath file
shorten command line option. The options are not visible for kotlin run configurations but can be accessed in a Java Application run application (with same options as a Kotlin version.). I think the broken option is the default so it would be much nicer if Ktor worked with that.
Expose LocationInfo in ktor locations
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1657
Subsystem
Server
Is your feature request related to a problem? Please describe.
We wanted our typed locations to be able to define a permission node / access level. We did this via an additional annotation but then we had to write an interceptor that would get the class annotated with @Location
from the request path and check for ours. The problem is that there doesn't seem to be any public access to the mapping done in Locations. Right now we're using a pretty-narly looking reflection solution that's less than ideal.
Describe the solution you'd like
Exposing io.ktor.locations.Locations$LocationInfo
and an immutable copy of io.ktor.locations.Locations.info
should be enough.
Motivation to include to ktor
I'm sure this could be a beneficial feature to a lot of users, even if it's not implemented in the way described above. It'll allow users to add additional properties to their routes and implement them in interceptors (and elsewhere) without having to do any reflection or map what is already mapped within ktor.locations.
InvocationTargetException when running the default ktor project
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1747
Hi all
I've created the Ktor project in Intellj with Ktor plugin, when running the application it shows the following error message:
020-03-23 20:31:20.192 [main] INFO Application - No ktor.deployment.watch patterns specified, automatic reload is not active
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at kotlin.reflect.jvm.internal.calls.CallerImpl$Method.callMethod(CallerImpl.kt:97)
at kotlin.reflect.jvm.internal.calls.CallerImpl$Method$Static.call(CallerImpl.kt:106)
at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:166)
at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:110)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.callFunctionWithInjection(ApplicationEngineEnvironmentReloading.kt:384)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:330)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:33)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1$$special$$inlined$forEach$lambda$1.invoke(ApplicationEngineEnvironmentReloading.kt:275)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1$$special$$inlined$forEach$lambda$1.invoke(ApplicationEngineEnvironmentReloading.kt:33)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:310)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:33)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:274)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:33)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartup(ApplicationEngineEnvironmentReloading.kt:290)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:272)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:125)
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:245)
The java version, that is installed on my computer:
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (Zulu 8.44.0.11-CA-linux64) (build 1.8.0_242-b20)
OpenJDK 64-Bit Server VM (Zulu 8.44.0.11-CA-linux64) (build 25.242-b20, mixed mode)
And here in the Application.kt
:
fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
install(Authentication) {
}
install(ContentNegotiation) {
gson {
}
}
val client = HttpClient() {
install(JsonFeature) {
serializer = GsonSerializer()
}
install(Logging) {
level = LogLevel.HEADERS
}
BrowserUserAgent() // install default browser-like user-agent
// install(UserAgent) { agent = "some user agent" }
}
runBlocking {
// Sample for making a HTTP Client request
/*
val message = client.post<JsonSampleClass> {
url("http://127.0.0.1:8080/path/to/endpoint")
contentType(ContentType.Application.Json)
body = JsonSampleClass(hello = "world")
}
*/
}
routing {
get("/") {
call.respondText("HELLO WORLD!", contentType = ContentType.Text.Plain)
}
get("/json/gson") {
call.respond(mapOf("hello" to "world"))
}
}
}
data class JsonSampleClass(val hello: String)
Thanks
Ktor 1.5.3 relies on kotlinx-html-jvm:0.7.3
Ktor 1.5.3 relies on kotlinx-html-jvm:0.7.3, but version 0.7.3 has not been published to any of the public maven repos.
Release 1.5.3
Fix flaky ProxyTest.testHttpProxy[CIO][jvm]
1.5.2
released 26th February 2021
Client
Changing `requestTimeoutMillis` in config of HttpTimeout feature doesn't change the CIO's timeout
Ktor 1.5.0
I am using CIO Engine which has a default timeout of 15000 milliseconds, however, I changed the timeout for a particular request using the HttpRequestBuilder.timeout
extension function, but I still get ConnectTimeoutException after 15000 milliseconds.
Gradle:
dependencies {
....
def ktorVersion = "1.5.0"
implementation "io.ktor:ktor-client-core:$ktorVersion"
implementation "io.ktor:ktor-client-cio:$ktorVersion"
implementation "io.ktor:ktor-client-serialization:$ktorVersion"
implementation "io.ktor:ktor-client-logging-jvm:$ktorVersion"
....
}
HttpClient:
HttpClient(CIO) {
defaultRequest {
host = HOST
header("Content-Type", "application/json")
}
Logging {
logger = object : Logger {
override fun log(message: String) {
Timber.i(message)
}
}
level = LogLevel.ALL
}
install(HttpTimeout)
}
Request:
httpClient.post("${HOST_URL}/users/new") {
timeout {
requestTimeoutMillis = 30000
}
body = requestBody
}
HTTP Client exception is masked by JobCancellationException with Ktor 1.5.0
STR: perform http GET with ktor Apache client to a resource which replies with truncated content
AR:
kotlinx.coroutines.JobCancellationException: Parent job is Cancelling; job=JobImpl{Cancelled}@eb5ab92
at kotlinx.coroutines.JobSupport.getChildJobCancellationCause(JobSupport.kt:711)
at kotlinx.coroutines.JobSupport.createCauseException(JobSupport.kt:717)
at kotlinx.coroutines.JobSupport.cancelMakeCompleting(JobSupport.kt:692)
at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:664)
at kotlinx.coroutines.JobSupport.parentCancelled(JobSupport.kt:634)
at kotlinx.coroutines.ChildHandleNode.invoke(JobSupport.kt:1465)
at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:1500)
at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:897)
at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:860)
at kotlinx.coroutines.JobSupport.cancelMakeCompleting(JobSupport.kt:693)
at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:664)
at kotlinx.coroutines.JobSupport.childCancelled(JobSupport.kt:648)
at kotlinx.coroutines.ChildHandleNode.childCancelled(JobSupport.kt:1466)
at kotlinx.coroutines.JobSupport.cancelParent(JobSupport.kt:358)
at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:897)
at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:860)
at kotlinx.coroutines.JobSupport.makeCompleting$kotlinx_coroutines_core(JobSupport.kt:803)
at kotlinx.coroutines.JobImpl.completeExceptionally(JobSupport.kt:1326)
at io.ktor.client.engine.apache.ApacheRequestProducer.failed(ApacheRequestProducer.kt:72)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:98)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:276)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 more
Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 178; received: 0)
at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:89)
at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:52)
at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:18)
at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1631)
at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:51)
at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
... 10 more
ER: ConnectionClosedException
is not masked
org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 178; received: 0)
at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:89)
at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:52)
at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:18)
at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1631)
at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:51)
at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
Core
Ktor Client CIO engine Jvm ignores Ciphersuites with key strength more than 128 bits.
The list contains TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 here :
https://github.com/ktorio/ktor/blob/master/ktor-network/ktor-network-tls/common/src/io/ktor/network/tls/CipherSuites.kt#L144-L152
but it is ignored by isSupported method here :
https://github.com/ktorio/ktor/blob/master/ktor-network/ktor-network-tls/jvm/src/io/ktor/network/tls/CipherSuitesJvm.kt#L9-L12
which makes these cipher suites unavailable to be used because of :
https://github.com/ktorio/ktor/blob/master/ktor-network/ktor-network-tls/jvm/src/io/ktor/network/tls/TLSHandshakeType.kt#L70
Session cookie with very long max age duration
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/893
When configuring a very very long session duration it may cause integer overflow. It need to be decided what should be done in this case:
- fail at attempt to set
duration
- bump
max-age
toInt.MAX_VALUE
orLong.MAX_VALUE
(long type requires changes, probably breaking changes)
This is related to #892.
Docs
Custom response validation is not running when default is disabled
[Outdated doc] Ktor quickstart contains outdated reference to version before 1.0
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1950
Ktor Version and Engine Used (client or server and name)
https://ktor.io/quickstart/quickstart/gradle.html#intellij-improve-the-app-with-the-application-object
Describe the bug
Ktor is currently at version 1.3.2, but in the quickstart docs it is mentioned that is not yet 1.0. Referenced line is: "Since Ktor is not yet 1.0, we have custom Maven repositories for distributing our early preview artifacts."
Expected behavior
Line should be corrected.
Docker doc is incorrect / does not work
https://ktor.io/docs/docker.html
The example does not work and is outdated.
Documentation search doesn't work on the website
It seems that the search functionality doesn't work in the docs.
Chrome Version 88.0.4324.192, macOS.
IntelliJ IDEA Plugin
Plugin does not generate a route for CSS DSL
Outdated Kotlin version in Ktor plugin
Ktor plugin generates a project with Kotlin 1.4.10
, but the latest version is 1.4.31
Wizard: Markdown Preview throws exceptions
Occurred 10 times in the last 5 or so minutes, just doing basic scrolling and picking and choosing of features.
java.lang.NoSuchMethodError: 'void org.intellij.plugins.markdown.ui.preview.MarkdownHtmlPanel.render()'
at io.ktor.initializr.intellij.features.KtorFeatureDetailsPageComponent$showKtorFeature$3$1.run(KtorFeatureDetailsPageComponent.kt:263)
at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313)
at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:776)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:746)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:969)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:839)
at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:449)
at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:808)
at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:448)
at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:781)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:502)
at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
IntelliJ IDEA 2021.1 Beta (Ultimate Edition)
Build #IU-211.6305.21, built on March 4, 2021
IntelliJ IDEA EAP User
Expiration date: April 4, 2021
Runtime version: 11.0.10+9-b1341.18 x86_64
VM: Dynamic Code Evolution 64-Bit Server VM by JetBrains s.r.o.
macOS 10.14.6
GC: G1 Young Generation, G1 Old Generation
Memory: 4096M
Cores: 8
Registry: ide.images.show.chessboard=true
Non-Bundled Plugins: com.jetbrains.darkPurpleTheme (1.2), org.nik.presentation-assistant (1.0.9), wu.seal.tool.jsontokotlin (3.6.1), org.asciidoctor.intellij.asciidoc (0.32.26), stardust (1.5.2105), com.jetbrains.intellij.api.watcher (6.46.0), intellij.ktor (1.5.1-eap-1)
Kotlin: 211-1.4.21-release-IJ6305.1
Samples
ktor-samples/client-mpp Android Studio doesn't see imports in iosMain
In ktor-samples/client-mpp
Android Studio doesn't see imported dependencies in the iosMain
sourceSet:
MacOS 11.1
Android Studio 4.1.1
XCode 12.3
ktor-samples/client-mpp doesn't build client-mpp-ios in Android Studio
Error message when trying to build the project
iOS configuration of ktor-samples/client-mpp
in Android Studio:
/Users/lukaszkalnik/projects/learning/ktor-samples/client-mpp/client-mpp-ios/ViewController.swift:3:8: error: no such module 'client_mpp'
import client_mpp
^
MacOS 11.1
Android Studio 4.1.1
XCode 12.3
Android Studio logs attached.
Server
Mandatory Path Segment parameter can be empty, if no explicit route with trailing / is defined
The route handler for /res/{parameter}
will now also match for /res/
. Before /res
and /res/
would match the "getAll" resource.
This means any "trailing" path segment parameter is implicitly treated as optional. So all handlers need to check if the parameter is empty string and then either
- redirect to
/res
- copy&paste the handler of
/res
to the empty string code branch - give a
404
or400
I guess "fixing" the behavior of /res
== /res/
in 1.5 is opinionated, so there should be a real option/flag to ignore trailing slashes.
Regression starting from 1.5 ( https://youtrack.jetbrains.com/issue/KTOR-372 ), works as expected in 1.4.3
The docs seem to be partially broken now for mandatory path segment parameters.
{param} matches a path segment and captures it as a parameter named param. This path segment is mandatory, but you can make it optional by adding a question mark: {param?}.
Allow to configure features for subroutes
Imagine that you have feature X and install it. Now you want to set up the feature config for the specific subroute.
In the current state, there is no generic way to do this. Users can change the config of the feature after install (ex, Authentication
feature) but only globally, and to do this, they need to understand how internal systems work.
Also, implementing a new DSL method in routing can break a priority of routes (see https://youtrack.jetbrains.com/issue/KTOR-613)
Suggested solution:
API
Not all the features can be dynamically configured in routing. For example, feature that starts a background process on install should not know anything about routing. To separate them in API we will create the new interface:
public interface RoutingScopedFeature<in TPipeline : Pipeline<*, ApplicationCall>, TConfiguration : Any, TFeature : Any> :
ApplicationFeature<TPipeline, TConfiguration, TFeature> {
...
/**
* Feature installation script
*/
public fun install(pipeline: TPipeline): TFeature
}
Note that the install
function doesn't have a configuration block as a parameter on the installation, because it can be modified in routing.
The new overload for the install
function will be introduced
public fun <Config, Feature : RoutingScopedFeature<*, Config, *>> Route.install(
feature: Feature,
configBuilder: Config.() -> Unit
)
To receive actual configuration inside the feature, the user will need to use the configurationBlock
property inside the interceptor.
pipeline.intercept(phase) {
val configuration = Configuration().apply(configurationBlock)
...
}
Usage example (Authentication
feature):
install(Authentication) {
form { validate { c -> UserIdPrincipal(c.name) } }
}
routing {
route(...) {
install(Authentication) {
form("2") {
validate { it.name.takeIf { it == "bbb" }?.let { UserIdPrincipal(it) } }
}
}
get("/") { }
}
}
Implementation
To check full (but draft) implementation with tests and migration of the Authentication
feature, please check the rsinukov/feature-config
branch.
Migration
We can gradually add dynamic configuration functionality to existing features by changing their base interface from ApplicationFeature
to DynamicConfigFeature
. The migration of each feature will require minimal efforts (can be seen in the rsinukov/feature-config
branch for the Authentication
feature).
CIO server always start on "0.0.0.0" - does not respect "connector" configuration
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1959
CIO Server ktor-server-cio:1.3.2-1.4-M2
Describe the bug
CIO server always start on "0.0.0.0" - does not respect "connector" configuration
To Reproduce
Steps to reproduce the behavior:
embeddedServer(CIO, applicationEngineEnvironment {
module { ... }
connector {
host = "192.168.0.1"
port = 8080
}
})
Server will start on "0.0.0.0:8080" instead of "192.168.0.1:8080"
Expected behavior
Server respect connector configuration.
server/netty: IllegalReferenceCountException
I got this random error out of nowhere. I'm not even using the Ktor server. It's just random bots on the web making HTTP calls to it.
I have no idea what request, call or response it was.
2021-01-19 21:21:41.833 E Unhandled exception caught for CoroutineName(call-handler)
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
at io.netty.handler.codec.http.DefaultFullHttpRequest.release(DefaultFullHttpRequest.java:103)
at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
at io.ktor.server.netty.NettyApplicationCall.finishComplete(NettyApplicationCall.kt:53)
at io.ktor.server.netty.NettyApplicationCall.finishSuspend(NettyApplicationCall.kt:46)
at io.ktor.server.netty.NettyApplicationCall$finishSuspend$1.invokeSuspend(NettyApplicationCall.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:227)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Installed features: CallLogging, Compression, DefaultHeaders, CORS, XForwardedHeaderSupport, WebSockets (unused)
Routes: 1x get
, 1x post
(both for GraphQL queries)
Digest authentication: cannot successfully pass authentication using curl or web browser
To reproduce run the server using the following code snippet:
fun main() {
embeddedServer(Netty, port = 8081) {
install(Authentication) {
digest("digest") {
val password = "123"
realm = "testrealm@host.com"
digestProvider { name, realm ->
"$name:$realm:$password".toByteArray()
}
}
}
routing {
authenticate("digest") {
get("/digest") {
call.respondText("ok")
}
}
}
}.start(wait = true)
}
Send the following HTTP request using curl
:
curl -v 'http://localhost:8081/digest' --digest -u guest:123
I expect successfully pass authentication and get a response with a status code 200 but instead, I get the following response:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="43f48979ca35e2a5", algorithm="MD5"
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8081/digest'
* Found bundle for host localhost: 0x7fbe62518750 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> Authorization: Digest username="guest", realm="testrealm@host.com", nonce="43f48979ca35e2a5", uri="/digest", response="ee8c56f444996c77cd1e5f7570035372", algorithm="MD5"
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="76d13a440ec99b1c", algorithm="MD5"
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0
Other
Test a POST with MultiPart using TestApplicationEngine does not success or fail
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1936
Ktor 1.3.2, TestApplicationEngine
When running a test that sends post with multipart it runs forever and never stops.
No problem when server really runs. It's just the test.
To Reproduce
Server:
fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
install(ContentNegotiation) {
}
routing {
post("/") {
val mp = try {
call.receiveMultipart()
} catch (e: Exception) {
call.application.log.error("Getting multipart error"")
null
}
if (mp == null) call.respond(HttpStatusCode.BadRequest, "Null value")
else call.respond(HttpStatusCode.OK,"OK")
}
}
}
Test:
internal class ApplicationKtTest {
private val boundary = "***bbb***"
private val sysTempDir = System.getProperty("java.io.tmpdir") ?: "/tmp"
private val multipart = listOf(PartData.FileItem({ byteArrayOf(1, 2, 3).inputStream().asInput() }, {}, headersOf(
HttpHeaders.ContentDisposition,
ContentDisposition.File
.withParameter(ContentDisposition.Parameters.Name, "file")
.withParameter(ContentDisposition.Parameters.FileName, "test.jpg")
.toString()
)))
@Test
fun testUploadApplication() = testApp {
handlePost("/", boundary, multipart).apply {
println("============= RESPONSE ====================")
println(response.content)
println("=================================")
}
}
}
private fun TestApplicationEngine.handlePost(uri: String,
boundary: String,
multipart: List<PartData.FileItem>,
setup: TestApplicationRequest.() -> Unit = {}
): TestApplicationCall {
return handleRequest(method = HttpMethod.Post, uri = uri) {
addHeader(HttpHeaders.ContentType,
ContentType.MultiPart.FormData.withParameter("boundary", boundary).toString()
)
setBody(boundary, multipart)
setup()
}
}
private fun testApp(callback: TestApplicationEngine.() -> Unit): Unit {
withTestApplication({ module(true) }, callback)
}
The test runs forever and gives no result. if addHeader
and setBody
are commented there is a response.
Expected behavior
The test should print OK
Repository
If you just want to clone a test repository.
Fix Dokka building for master
The list allCompileKotlinTasks
was removed due to refactoring
Native does not build on linux machine
Need to install missing dependencies:
libncurses5 libncursesw5 libtinfo5
Fix existing samples
The project doesn't compile
Fix testUnsafeHeaders for Js platform
Fix tests failing due to compiler error. The following error is in references
Fix flaky ClientSocketTest.testSelfConnect[jvm]
https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreMacOSJava11/21952
https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreWindowsJava11/21949
https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreMacOSJava8/21953
https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreWindowsJava8/21951
Default Headers feature adds duplicated Server header
Crash with Firebase Performance in iOS
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1040
Ktor Version
1.1.3
Ktor Engine Used(client or server and name)
Client iOS with default engine
Feedback (Bug)
When adding Firebase Performance to an iOS application, the app crashes when making a request with Ktor.
I've forked the sample mmp project and added the Firebase Performance library. The application crashes due to Firebase Performance. It might be because Firebase intercepts all network requests and it might be incompatible with Ktor.
You can find the sample mmp client crashing with Firebase Performance here https://github.com/eduardbosch/ktor-samples/commit/35bb435b4138ef966eeb9c44cbaf632bfaaeb305
Remember to add the GoogleServices-Info.plist to prevent Firebase to crash when it is initialised.
I hope this can help Ktor! 🎉
CIO Server Engine fails for requests with more than 32 headers
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1095
Ktor Version
1.1.5
Ktor Engine Used(client or server and name)
CIO server
JVM Version, Operating System and Relevant Context
1.8.0_131
Feedback
When using the CIO server engine, requests with more than 32 headers are rejected with error 400 and no further diagnostic.
I used the debugger to further track the cause of this error and found that during header parsing an exception is thrown:
kotlin.NotImplementedError: An operation is not implemented: Implement headers overflow
This is due to the following declaration in file HttpHeaderMap.kt:
package io.ktor.http.cio
import io.ktor.http.cio.internals.*
import io.ktor.util.*
import kotlinx.io.pool.*private const val EXPECTED_HEADERS_QTY = 32
which effectively limits the number of headers to 32.
This error does not appear with the Jetty engine.
Release 1.5.2
1.5.1
released 28th January 2021
Client
http client (native macos_x64) v0.9.5-rc13 Memory leaks found
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/593
Whenever I try and do a simple http request on macOs the program freezes and reports a memory leak:
an example where I publish the latest eap13 branch commit to localMaven which is tagged (v0.9.5-rc13):
buildscript {
repositories {
jcenter()
maven { url 'https://plugins.gradle.org/m2/' }
maven { url 'https://dl.bintray.com/jetbrains/kotlin-native-dependencies' }
maven {url 'http://dl.bintray.com/kotlin/kotlin-dev'}
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-native-gradle-plugin:0.9.2-dev-4008"
}
}
allprojects {
project.ext {
//make extras set inside buildscript available in all modules
kotlin_version = '1.3.0-rc-51'
kotlin_coroutines_version = '0.26.1-eap13'
}
}
apply plugin: 'kotlin-platform-native'
repositories {
mavenLocal()
jcenter()
}
components.main {
targets = ["macos_x64"]
outputKinds = [EXECUTABLE]
dependencies {
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core-native:0.26.1-eap13")
//you need to clone ktor and publish to local maven
implementation "io.ktor:ktor-client-ios:0.9.4-SNAPSHOT"
}
}
example program:
import kotlinx.coroutines.runBlocking
import io.ktor.client.engine.ios.*
import io.ktor.client.*
import io.ktor.http.*
import io.ktor.client.request.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import platform.darwin.*
internal val ApplicationDispatcher: CoroutineDispatcher = NsQueueDispatcher(dispatch_get_main_queue())
fun main(args: Array<String>) = runBlocking<Unit> {
val api = ApplicationApi()
api.about {
println(it)
}
delay(3000L)
return@runBlocking
}
internal class NsQueueDispatcher(
private val dispatchQueue: dispatch_queue_t
) : CoroutineDispatcher() {
override fun dispatch(context: CoroutineContext, block: Runnable) {
dispatch_async(dispatchQueue) {
block.run()
}
}
}
class ApplicationApi {
private val client = HttpClient()
fun about(callback: (String) -> Unit) {
launch(ApplicationDispatcher) {
val result: String = client.get {
url {
protocol = URLProtocol.HTTPS
port = 443
host = "tools.ietf.org"
encodedPath = "rfc/rfc1866.txt"
}
}
callback(result)
}
}
}
when run:
/opt/teamcity-agent/work/4d622a065c544371/runtime/src/main/cpp/Memory.cpp:1124: runtime
assert: Memory leaks found
Abort trap: 6
Remove dependency on 'text-encoding' npm package
It's deprecated and looks like TextEncoder and TextDecoder work on majority of environments.
Logging tests fails due to floating log entries
Sometimes a log entry may appear at the "wrong" place (before or after expected) so the testLogResponseWithException
and siblings are flaky.
CIO native selector doesn't select new descriptors
CIO native selector doesn't select new descriptors and unable to stop when closing because cancelled interests are not processed as well.
OkHTTP client engine tries to close the connection twice during the closing handshake
Here's how I noticed:
When I cancel the coroutine containing the WebSocket connection, KTor closes the connection with a 1011
. If the server then responds by closing the connection without a status code, KTor will get a 1005
and it'll try to pass it to OkHTTP.
However, 1005
is a reserved status, so OkHTTP refuses it when Ktor passes it in the following lines: https://github.com/ktorio/ktor/blob/6053fa44a61fdfc5474ff909c0224e56877b5da1/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L62
Here's an example of a traceback you'd get:
java.util.concurrent.CancellationException: ActorCoroutine was cancelled
at kotlinx.coroutines.ExceptionsKt.CancellationException(Exceptions.kt:22)
at kotlinx.coroutines.channels.ActorCoroutine.onCancelling(Actor.kt:134)
at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:916)
at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:875)
at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:840)
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.IllegalArgumentException: Code 1005 is reserved and may not be used.
at okhttp3.internal.ws.WebSocketProtocol.validateCloseCode(WebSocketProtocol.kt:134)
at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:435)
at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:427)
at io.ktor.client.engine.okhttp.OkHttpWebsocketSession$outgoing$1.invokeSuspend(OkHttpWebsocketSession.kt:62)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
I think there are technically two different issues happening at the same time:
- Ktor is trying to close the connection twice. The first time with a
1011
status (which I can confirm in the server logs) and the second time with a1005
(because the server didn't provide a closing status code). - Ktor is trying to close the connection with a reserved status code.
Java client logging tests are fluky
LoggingTest.testRequestContentTypeInLog
and LoggingTest.testRequestAndResponseBody
are fluky with Java client.
Quick investigation after fixing KTOR-1598 showed that the client dispatcher gets shutdown too early so the response body handling coroutines was unable to resume due to RejectedExecutionException
.
The reason is that coroutines jobs structure was wrong and the client infrastructure was waiting for the wrong job instance.
Response channel is always cancelled with Logging feature
val channel = HttpClient {
Logging {
}
}.get<ByteReadChannel>("/test")
channel.isClosedForRead == true
// channel is always closed with exception here
HttpTimeoutTest.testConnect are flaky
Tests HttpTimeoutTest.testConnectTimeout
and testConnectTimeoutPerRequestAttributes
somtimes may fail because of the unexpected exception text that is heavily system-dependant.
This is not a test issue but a real bug in okhttp integration
HttpTimeout.testSocketTimeoutWriteFail is flaky
Tests testSocketTimeoutWriteFail* sometimes may pass because of missing failures on Apache.
The possible reason is that sleep duration is only one second while timeouts are checked by apache every selection (that is also 1sec by default). So if the test server is too fast or selection is a little bit slower, we may get a response too early so no timeout happens in this case.
Jetty: requests to resources, that doesn't respond with HTTP/2, lead to unexpected behaviour
To reproduce run the following code:
fun main() {
val client = HttpClient(Jetty) {}
runBlocking {
val r = client.get<String>("http://www.google.com")
println(r)
}
}
As a result, org.eclipse.jetty.io.EofException
will be thrown:
Exception in thread "main" org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:259)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:543)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$jettyRequest$1.invoke(JettyHttpRequest.kt:40)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$jettyRequest$1.invoke(JettyHttpRequest.kt)
at io.ktor.client.engine.jetty.UtilsKt.withPromise(utils.kt:14)
at io.ktor.client.engine.jetty.JettyHttpRequestKt.executeRequest(JettyHttpRequest.kt:39)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$1.invokeSuspend(JettyHttpRequest.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.nio.channels.AsynchronousCloseException
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
... 18 more
If I change the URL to http://example.com
then the app will hang indefinitely, here is a stacktrace:
"main" #1 prio=5 os_prio=0 cpu=337.00ms elapsed=56.16s tid=0x00007f8e98018800 nid=0x79547 waiting on condition [0x00007f8e9eb77000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x00000006890f1518> (a kotlinx.coroutines.BlockingCoroutine)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:87)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at MainKt.main(main.kt:86)
at MainKt.main(main.kt)
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.36ms elapsed=56.11s tid=0x00007f8e98311000 nid=0x79550 waiting on condition [0x00007f8e80382000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.7/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@11.0.7/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.7/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=1.63ms elapsed=56.11s tid=0x00007f8e98315000 nid=0x79551 in Object.wait() [0x00007f8e80281000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.7/Native Method)
- waiting on <0x00000006890f2008> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x00000006890f2008> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.7/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.22ms elapsed=56.10s tid=0x00007f8e9832a000 nid=0x79552 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=497.80ms elapsed=56.10s tid=0x00007f8e9832c000 nid=0x79553 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=306.59ms elapsed=56.10s tid=0x00007f8e9832e000 nid=0x79554 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #7 daemon prio=9 os_prio=0 cpu=6.70ms elapsed=56.09s tid=0x00007f8e98330000 nid=0x79555 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #8 daemon prio=8 os_prio=0 cpu=0.37ms elapsed=56.03s tid=0x00007f8e98381800 nid=0x79556 in Object.wait() [0x00007f8e47efd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.7/Native Method)
- waiting on <0x00000006890f27e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x00000006890f27e0> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@11.0.7/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
at jdk.internal.misc.InnocuousThread.run(java.base@11.0.7/InnocuousThread.java:134)
"Monitor Ctrl-Break" #9 daemon prio=5 os_prio=0 cpu=8.39ms elapsed=55.91s tid=0x00007f8e98695800 nid=0x79557 runnable [0x00007f8e479d4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.7/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.7/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.7/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.7/SocketInputStream.java:140)
at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.7/StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.7/StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(java.base@11.0.7/StreamDecoder.java:178)
- locked <0x00000006890f4d00> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(java.base@11.0.7/InputStreamReader.java:185)
at java.io.BufferedReader.fill(java.base@11.0.7/BufferedReader.java:161)
at java.io.BufferedReader.readLine(java.base@11.0.7/BufferedReader.java:326)
- locked <0x00000006890f4d00> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(java.base@11.0.7/BufferedReader.java:392)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:61)
"Service Thread" #10 daemon prio=9 os_prio=0 cpu=0.04ms elapsed=55.91s tid=0x00007f8e98696800 nid=0x79558 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"ktor-jetty-dispatcher-worker-1" #11 daemon prio=5 os_prio=0 cpu=230.54ms elapsed=55.68s tid=0x00007f8e989de000 nid=0x7955a waiting on condition [0x00007f8e46ec0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:357)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"ktor-jetty-dispatcher-worker-2" #12 daemon prio=5 os_prio=0 cpu=9.27ms elapsed=55.68s tid=0x00007f8e989df000 nid=0x7955b waiting on condition [0x00007f8e46dbf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:357)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"ktor-jetty-client-qtp-13" #13 prio=5 os_prio=0 cpu=6.95ms elapsed=55.47s tid=0x00007f8e340da000 nid=0x7955d runnable [0x00007f8e46575000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.wait(java.base@11.0.7/Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.7/EPollSelectorImpl.java:120)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.7/SelectorImpl.java:124)
- locked <0x000000069f414300> (a sun.nio.ch.Util$2)
- locked <0x000000069f4142a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@11.0.7/SelectorImpl.java:141)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:472)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:409)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$65/0x0000000800176c40.run(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-14" #14 prio=5 os_prio=0 cpu=126.50ms elapsed=55.47s tid=0x00007f8e340dc000 nid=0x7955e waiting on condition [0x00007f8e46474000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-15" #15 prio=5 os_prio=0 cpu=0.27ms elapsed=55.47s tid=0x00007f8e340dd800 nid=0x7955f waiting on condition [0x00007f8e46373000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069fc076c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2211)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:309)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:379)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-16" #16 prio=5 os_prio=0 cpu=2.68ms elapsed=55.46s tid=0x00007f8e340df800 nid=0x79560 waiting on condition [0x00007f8e46272000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-17" #17 prio=5 os_prio=0 cpu=0.10ms elapsed=55.46s tid=0x00007f8e340e1000 nid=0x79561 waiting on condition [0x00007f8e46171000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-18" #18 prio=5 os_prio=0 cpu=0.07ms elapsed=55.46s tid=0x00007f8e340e3000 nid=0x79562 waiting on condition [0x00007f8e46070000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-19" #19 prio=5 os_prio=0 cpu=0.07ms elapsed=55.46s tid=0x00007f8e340e5000 nid=0x79563 waiting on condition [0x00007f8e45f6f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"ktor-jetty-client-qtp-20" #20 prio=5 os_prio=0 cpu=0.06ms elapsed=55.46s tid=0x00007f8e340e7000 nid=0x79564 waiting on condition [0x00007f8e45e6e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"Scheduler-779376001-1" #21 prio=5 os_prio=0 cpu=10.86ms elapsed=55.42s tid=0x00007f8e34117000 nid=0x79565 waiting on condition [0x00007f8e45b6d000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x000000069f415c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.7/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.7/ScheduledThreadPoolExecutor.java:1170)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.7/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.7/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
"Attach Listener" #22 daemon prio=9 os_prio=0 cpu=8.56ms elapsed=55.08s tid=0x00007f8e4805c800 nid=0x79567 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=0 cpu=16.20ms elapsed=56.11s tid=0x00007f8e98309000 nid=0x7954f runnable
"GC Thread#0" os_prio=0 cpu=22.88ms elapsed=56.16s tid=0x00007f8e98041800 nid=0x7954a runnable
"GC Thread#1" os_prio=0 cpu=22.20ms elapsed=55.66s tid=0x00007f8e5c001000 nid=0x7955c runnable
"G1 Main Marker" os_prio=0 cpu=0.51ms elapsed=56.15s tid=0x00007f8e980cc800 nid=0x7954b runnable
"G1 Conc#0" os_prio=0 cpu=0.07ms elapsed=56.14s tid=0x00007f8e980ce000 nid=0x7954c runnable
"G1 Refine#0" os_prio=0 cpu=0.50ms elapsed=56.13s tid=0x00007f8e9823b800 nid=0x7954d runnable
"G1 Young RemSet Sampling" os_prio=0 cpu=10.66ms elapsed=56.13s tid=0x00007f8e9823d800 nid=0x7954e runnable
"VM Periodic Task Thread" os_prio=0 cpu=42.21ms elapsed=55.91s tid=0x00007f8e98698000 nid=0x79559 waiting on condition
JNI global refs: 14, weak refs: 0
If I make the same requests with curl: curl --http2-prior-knowledge http://example.com
then I get an understandable error:
curl: (16) Error in the HTTP2 framing layer
Java client freeze
Java client tests may freeze from time to time on CI. Debugging showed up that it's a bug.
"Unfinished workers detected" using client on native
To reproduce run ./gradlew runDebugExecutableMacosX64
on the attached project.
As a result, I get unexpected Unfinished workers detected, 1 workers leaked!
warning and the program finishes with 134 exit code.
As a workaround, I can add Platform.isMemoryLeakCheckerActive = false
line to prevent this behavior.
Here is a code example that after execution reports Unfinished workers detected, 100 workers leaked!
warning:
fun main() {
runBlocking {
repeat(100) {
val client = HttpClient(Curl)
val data = client.get<String> {
url("http://google.com")
}
println(data)
client.close()
}
}
}
ResponseException is no longer serializable starting from 1.4.0 (breaking change)
Previously a ResponseException could be serialized. See: https://github.com/ktorio/ktor/issues/1256. The Transient annotation was removed in a recent commit for the HttpResponse
field. Now an exception is thrown when serializing a ResponseException.
Stack trace:
io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
java.io.NotSerializableException: io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
at ExceptionsTest.serialize(ExceptionsTest.kt:26)
at ExceptionsTest.testResponseExceptionSerializable(ExceptionsTest.kt:17)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Here is a test to reproduce the issue:
import io.ktor.client.call.*
import io.ktor.client.features.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.util.date.*
import io.ktor.utils.io.*
import java.io.*
import kotlin.coroutines.*
import kotlin.test.*
class ExceptionsTest {
@Test
fun testResponseExceptionSerializable() {
val exception = createResponseException()
val serialized = serialize(exception)
val deserialized = deserialize(serialized)
deserialized as ResponseException
}
private fun serialize(obj: Any): ByteArray {
val b = ByteArrayOutputStream()
val o = ObjectOutputStream(b)
o.writeObject(obj)
return b.toByteArray()
}
private fun deserialize(bytes: ByteArray): Any? {
val b = ByteArrayInputStream(bytes)
val o = ObjectInputStream(b)
return o.readObject()
}
}
private fun createResponseException(): ResponseException = ResponseException(object : HttpResponse() {
override val call: HttpClientCall
get() = TODO("Not yet implemented")
override val status: HttpStatusCode
get() = TODO("Not yet implemented")
override val version: HttpProtocolVersion
get() = TODO("Not yet implemented")
override val requestTime: GMTDate
get() = TODO("Not yet implemented")
override val responseTime: GMTDate
get() = TODO("Not yet implemented")
override val content: ByteReadChannel
get() = TODO("Not yet implemented")
override val headers: Headers
get() = TODO("Not yet implemented")
override val coroutineContext: CoroutineContext
get() = TODO("Not yet implemented")
override fun toString(): String = "FakeCall"
}, cachedResponseText = "Fake text")
ClosedReceiveChannelException when making request with CIO engine using a proxy to https
version: 1.4.3
when use a http proxy, get the exception.
use no proxy, it's ok.
code
val httpClient = HttpClient(CIO) {
engine {
proxy = httpProxy
}
}
val c : String = httpClient.get(url)
build.gradle.kts
// ...
dependencies {
implementation("org.jetbrains.kotlinx", "kotlinx-serialization-json", "1.0.1")
implementation("io.ktor", "ktor-client-core", "1.4.3")
implementation("io.ktor", "ktor-client-cio", "1.4.3")
implementation("io.ktor", "ktor-client-serialization-jvm", "1.4.3")
implementation("org.jetbrains.kotlinx", "kotlinx-coroutines-core", "1.4.2")
// test
}
exception:
Channel was closed
kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
(Coroutine boundary)
at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)
at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:76)
at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:81)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
(Coroutine creation stacktrace)
at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30)
at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:128)
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:92)
at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:123)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at LoginTest$testLogin$1.invokeSuspend(LoginTest.kt:18)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at mytest(...)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:205)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:201)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:129)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at mytest(...)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at mytest(...)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at mytest(...)
Caused by: kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
ktor-client-apache: thread stuck in ByteBufferChannel.readRemainingSuspend
In production I've got two threads stuck in 100% CPU usage due to Ktor.
The server is running without debugging agent, so I cannot debug the exact scenario just yet. I'll restart it today with debugging agent running so that I can investigate further the next time the issue occurs.
When creating thread dumps I get one of the following two outputs. Note the high CPU usage and that the threads seem to be stuck in ByteBufferChannel.readRemainingSuspend
.
"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51203112.18ms elapsed=53179.31s tid=0x00007f6fc0052510 nid=0xa9b2 runnable [0x00007f6f0ebe9000]
java.lang.Thread.State: RUNNABLE
at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible$default(ByteBufferChannel.kt:2678)
at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:2231)
at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51253887.85ms elapsed=53230.30s tid=0x00007f6fc0052510 nid=0xa9b2 runnable [0x00007f6f0ebe9000]
java.lang.Thread.State: RUNNABLE
at io.ktor.utils.io.core.internal.UnsafeKt.prepareWriteHead(Unsafe.kt:160)
at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:3060)
at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Circular reference for SocketException and StackOverflowError when using SLF4J logger
I upgraded my project to ktor 1.4.1 today.
I found an issue when I ran my unit test.
My logger failed to log the SocketTimeoutException.
Sample code:
fun main() {
val httpClient = HttpClient(Apache) {
install(HttpTimeout) {
socketTimeoutMillis = 3000
}
}
runBlocking {
try {
val get = httpClient.get<String>("https://httpbin.org/delay/10")
} catch (e: Exception) {
e.printStackTrace()
LoggerFactory.getLogger("main").error("test", e)
}
}
}
Output:
io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:49)
at io.ktor.client.engine.apache.ApacheHttpRequestKt$sendRequest$1.invokeSuspend(ApacheHttpRequest.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
at io.ktor.client.engine.apache.ApacheResponseConsumer.failed(ApacheResponseConsumer.kt:73)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:99)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:834)
[CIRCULAR REFERENCE:io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms]
Caused by: java.net.SocketTimeoutException: 3,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
... 11 more
Exception in thread "main" java.lang.StackOverflowError
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:590)
at java.base/java.lang.reflect.Method.invoke(Method.java:558)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:66)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:72)
Core
Fix socket self connect problem
I already made a pr on github: https://github.com/ktorio/ktor/pull/2237
Subsystem
ktor-network-jvm
Motivation
TCP has a well known self-connect problem, which client can connect to the client itself without any program listen on the port.
Reproduce Code :
fun testSelfConnect() {
runBlocking {
// Find a port that would be used as a local address.
val port = getAvailablePort()
val tcpSocketBuilder = aSocket(ActorSelectorManager(Dispatchers.IO)).tcp()
// Try to connect to that address repeatedly.
for (i in 0 until 100000) {
try {
val socket = tcpSocketBuilder.connect(InetSocketAddress("127.0.0.1", port))
println("connect to self succeed: ${socket.localAddress} to ${socket.remoteAddress}")
System.`in`.read()
break
} catch (ex: Exception) {
// ignore
}
}
}
}
// since new linux kernel version introduce a feature, new bind port number will always be odd number
// and connect port will always be even, so we find a random even port with while loop
// see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=07f4c90062f8fc7c8c26f8f95324cbe8fa3145a5
private fun getAvailablePort(): Int {
while (true) {
val port = ServerSocket().apply {
bind(InetSocketAddress("127.0.0.1", 0))
close()
}.localPort
if (port % 2 == 0) {
return port
}
try {
// try bind the next even port
ServerSocket().apply {
bind(InetSocketAddress("127.0.0.1", port + 1))
close()
}
return port + 1
} catch (ex: Exception) {
// ignore
}
}
}
Running the above code whill result in a tcp connection with src addr equal to dst addr and src port equal to dst port.
If this happens, client will behave abnormal because it connect to the wrong server,server will bind fail on this machine.
Other programming language also have this problem, eg golang: https://golang.org/issue/2690
Fix this problem is easy, just check if the local and remote socket pair are equal.
Upgrade kotlin to 1.4.21
testTimeoutCancelsWhenParentScopeCancels is flaky
StartTimeoutTest.testTimeoutCancelsWhenParentScopeCancels is flaky: 8 failures in almost 1000 builds
The reason is that the timeout coroutine may trigger too fast after changing the test clock but before the parent cancellation
Incorrect encoding function used for URL path by URLBuilder
If you use URLBuilder
to create a URL, it incorrectly encodes the path. Per https://tools.ietf.org/html/rfc3986#section-2 3.3, a The path is terminated by the first question mark ("?") or number sign ("#") character, or by the end of the URI.
. However the #
symbol isn't encoded by the path
function.
val rawPath = "id+test&test~test#test"
val urlString = url {
path(rawPath)
}
// urlString = http://localhost/id+test&test~test#test
If you look at the source of the path
function you will see it uses <string>.encodeURLQueryComponent()
which is the incorrect function. If it used the <string>.encodeURLPath()
the #
would be correctly encoded as id+test&test~test%23test
.
public fun path(components: List<String>): URLBuilder {
encodedPath = components.joinToString("/", prefix = "/") { it.encodeURLQueryComponent() }
return this
}
Unable to catch socket exceptions
Exceptions thrown from the UDP sockets cannot be caught.
Here is a test that currently fails:
import io.ktor.network.selector.*
import io.ktor.network.sockets.*
import io.ktor.util.network.*
import io.ktor.utils.io.core.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import kotlin.io.use
import kotlin.test.*
class SocketExceptionIssue : CoroutineScope {
private val testJob = Job()
private val selector = ActorSelectorManager(Dispatchers.Default + testJob)
override val coroutineContext: CoroutineContext
get() = testJob
@AfterTest
fun tearDown() {
testJob.cancel()
selector.close()
}
@Test
fun testBroadcastFails(): Unit = runBlocking {
withTimeout(15000) {
// The following code in the assertFails lambda should throw a SocketException.
assertFails {
aSocket(selector)
.udp()
.bind {
// Do not set broadcast to true so exception is thrown.
}
.use { socket ->
// Send a broadcast which will fail because broadcasts are not enabled on this socket.
socket.send(
Datagram(
packet = buildPacket { writeText("0123456789") },
address = NetworkAddress("255.255.255.255", 56700)
)
)
}
}
}
}
}
When running this test the logs show the following:
> Task :ktor-network:jvmTest
Exception in thread "DefaultDispatcher-worker-1 @coroutine#3" java.net.SocketException: Permission denied
at java.base/sun.nio.ch.DatagramChannelImpl.send0(Native Method)
at java.base/sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:584)
at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:546)
at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:529)
at io.ktor.network.sockets.DatagramSocketImpl.sendImpl(DatagramSocketImpl.kt:98)
at io.ktor.network.sockets.DatagramSocketImpl$sender$1.invokeSuspend(DatagramSocketImpl.kt:35)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Expected an exception to be thrown, but was completed successfully.
java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
at org.junit.Assert.fail(Assert.java:88)
at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
(Coroutine boundary)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
(Coroutine creation stacktrace)
at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at io.ktor.network.sockets.tests.SocketExceptionIssue.testBroadcastFails(SocketExceptionIssue.kt:30)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
at org.junit.Assert.fail(Assert.java:88)
at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:81)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:56)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
... 50 more
io.ktor.network.sockets.tests.SocketExceptionIssue[jvm] > testBroadcastFails[jvm] FAILED
java.lang.AssertionError at SocketExceptionIssue.kt:30
Caused by: java.lang.AssertionError at SocketExceptionIssue.kt:30
1 test completed, 1 failed
You can see the SocketException in the log and that the test (with assertFails) is unable to catch it.
The following issue looks related: https://youtrack.jetbrains.com/issue/KTOR-440
Here is a fork of Ktor with the test added so you can reproduce the issue: https://github.com/Thomas-Vos/ktor/tree/socket_exception_issue (see last commit for added test). The fork is based on the 1.5.0 branch.
Docs
Add local development guide to the contributing document
Add following information to the CONTRIBUTING document:
- How to publish artifacts locally - Maven Local or includeBuild...
- How to set up a sample project for local development - Referencing Ktor locally
- How to debug Ktor in IDEA
- How to write integration tests - Ask Team for that.
And maybe some additional information that will make the life of a contributor easier.
start.ktor.io - Incorrect import for websockets for ktor 1.2.4
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/225
Generating files using Ktor 1.2.4
and Websocket HTTP Client Support
dependency makes it so the project will not build.
build.gradle
file specifies compile "io.ktor:ktor-client-websocket:$ktor_version"
which will not work since ktor-client-websocket
does not have v1.2.4 in the repositories.
Client logging docs don't mention all required dependencies
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/185
https://ktor.io/clients/http-client/features/logging.html clearly mentions io.ktor:ktor-client-logging
artifact, but after adding that artifact and copy-pasting the code, it still doesn't compile because it also needs io.ktor:ktor-client-logging-jvm
for example.
After adding that, it compiles, but then I need the hidden knowledge of https://ktor.io/servers/logging.html#providers (which is Server docs) to add something like implementation("org.slf4j:slf4j-simple:1.7.26")
Out of date self-signed-certificate documentation
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/255
https://github.com/ktorio/ktorio.github.io/blob/master/servers/self-signed-certificate.md
The documentation reads
To create a self-signed certificate using Ktor, you have to call the generateCertificate function.
io.ktor.network.tls.certificates.generateCertificate(File("mycert.jks"))
With
implementation "io.ktor:ktor-network-tls:$ktor_version"
where ktor_version is 1.3.0
, there is no generateCertificate function defined in io.ktor.network.tls.certificates
import io.ktor.network.tls.certificates.generateCertificate
is an invalid import.
Custom JSON mapping with Jackson
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1208
Ktor Version
1.2.2
Feedback
I'm using Jackson mapping in the Client (here: https://ktor.io/clients/http-client/features/json-feature.html), but there seems to be no way to configure the mapper. Normally, I would just include a custom-configured object mapper (the same way I do for server component), but here it's just a single-constructor class tha explicitly calls for the default mapper inside.
Can we get a configuration param for the object mapper? Alternatively, a builder to specify it.
New documentation lacks artifacts information for Gradle and Maven
For example, the old documentation page about Jackson feature has ARTIFACT
block but the corresponding new documentation page has not.
Missing dependency information the Client Auth topic
Add information about required artifacts to the WebSockets topic
Serialization for client section does not explain how to use it
Serialization for client section does not explain how to use it and it doesn't tell anything about required headers.
https://ktor.io/clients/http-client/features/json-feature.html -- This is the page that describes the Json feature. It only teaches how to install the feature (and reader assumes that it will work out of the box right after installation, and any data will be automatically serialized which is not quite true because it also requires contentType
to be set).
This implies that user can just write something like:
@Serializable
data class HelloWorld(val text: String = "Hello world")
val client = HttpClient(Apache) {
install(JsonFeature) {
serializer = KotlinxSerializer()
}
}
client.post<Unit> {
url("http://127.0.0.1:8080/")
body = HelloWorld()
}
But this is not true:
Exception in thread "main" java.lang.IllegalStateException: Fail to serialize body. Content has type: class HelloWorld, but OutgoingContent expected.
If you expect serialized body, please check that you have installed the corresponding feature(like `Json`) and set `Content-Type` header.
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:79)
at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
at io.ktor.client.HttpClient.execute(HttpClient.kt:185)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at MainKt$main$1.invokeSuspend(main.kt:49)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at MainKt.main(main.kt:19)
It also requires content type (ex., contentType(ContentType.Application.Json)
) to be set. This information is not discoverable and should be documented.
"Testing Http Client" docs page contains artifact name with -native suffix
To reproduce add api "io.ktor:ktor-client-mock-native:$ktor_version"
artifact, that is described on the Testing Http Client (MockEngine) docs page, to the Gradle build file. If ktor_version
is 1.4.0
then the build fails with unexpected error:
Could not find io.ktor:ktor-client-mock-native:1.4.0.
The migration guide for 1.4.0
explains that behavior.
Missing dependency information the Authentication and Authorization topic
"Using a Self-Signed Certificate" docs provide wrong dependency for 1.3.x
https://ktor.io/servers/self-signed-certificate.html says :
- This function is defined in the method io.ktor.network.tls.certificates.generateCertificate in the artifact io.ktor:ktor-network-tls:$ktor_version.
But it is wrong for 1.3.x Ktor versions (see https://github.com/ktorio/ktor/issues/1612). The workaround is to use "io.ktor:ktor-network-tls-certificates:$ktor_version" for 1.3.x.
Samples also need to be fixed
ktor server documentation is returning 404
when hitting https://ktor.io/docs/servers-testing.html getting 404
[docs] Wrong shadow plugin version in Fat JAR docs
The fatjar documentation page https://ktor.io/docs/fatjar.html#fat-jar-gradle recommends using shadow plugin 2.0.4.
But I faced this error:
Task :shadowJar FAILED
A problem was found with the configuration of task ':shadowJar' (type 'ShadowJar').
No value has been specified for property 'mainClassName'.
I fixed this by updating the plugin from 2.0.4 to 6.1.0 (latest)
Example project: https://github.com/egorklimov/ktor-fatjar-issue
Add "Edit on GitHub" button to docs
Like we have on https://kotlinlang.org/docs/home.html, to encourage external contributors to provide documentation content and fixes.
IntelliJ IDEA Plugin
Ktor wizard generates old kotlin version
Validate for Empty Input Fields in New Project Wizard
The plugin allows using empty fields for project.name
Server
Adding existing dropwizard metrics registry to Ktor
I saw that I could add dropwizard-metrics to Ktor and get it to create metrics for resources and such, really neat! In my app I already have a metrics registry that I have added some metric sets for threads, memory and such to. When I add my existing registry to Ktor like this:
install(DropwizardMetrics) {
registry = metricRegistry
}
When starting the app I realize that Ktor registers the same metric sets and it is causing issues. Like this:
Failed to start WebServer, due to: 'A metric named jvm.memory.heap.committed already exists', retrying
How do I register my existing metrics registry and avoid having Ktor register the same metrics again?
Request parameters should have name
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1868
Ktor Version and Engine Used (client or server and name)
io.ktor:ktor-server-cio:1.3.2
Describe the bug
Missing header name should fail(see https://tools.ietf.org/html/rfc2616#section-4.2)
To Reproduce
Steps to reproduce the behavior:
Run code:
import io.ktor.http.cio.parseRequest
import io.ktor.utils.io.ByteReadChannel
import kotlinx.coroutines.runBlocking
fun main() {
test(
"""
GET / HTTP/1.1
Host: www.example.com
: value
""".trimIndent()
)
}
fun test(http: String) {
val request = runBlocking { parseRequest(ByteReadChannel(http)) }!!
println("method=${request.method};version=${request.version};uri={${request.uri}};headers=[${request.headers}]")
}
Expected behavior
An error expected - exception or null result.
CORS doesn't reject bad headers
After introducing accepted headers predicate, CORS started to accept non-simple headers by default. It is not a security issue because clients will not pass it anyway in spite of success.
Status-code must be 3-digit
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1882
Ktor Version and Engine Used (client or server and name)
io.ktor:ktor-server-cio:1.3.2
Describe the bug
Status-code must be 3-digit(see https://tools.ietf.org/html/rfc2616#section-6.1.1)
To Reproduce
Steps to reproduce the behavior:
Run code:
import io.ktor.http.cio.parseResponse
import io.ktor.utils.io.ByteReadChannel
import kotlinx.coroutines.runBlocking
fun main() {
listOf(
"""
HTTP/1.1 0 OK
""".trimIndent(),
"""
HTTP/1.1 10 OK
""".trimIndent(),
"""
HTTP/1.1 1000 OK
""".trimIndent(),
"""
HTTP/1.1 10000 OK
""".trimIndent()
).forEach { test(it) }
}
fun test(http: String) {
val response = runBlocking { parseResponse(ByteReadChannel(http)) }!!
println("status=${response.status};statusText={${response.statusText}};headers=[${response.headers}];version=${response.version}")
}
Expected behavior
An error expected - exception or null result.
Server losing channel exceptions at receive
If the input channel is broken due to some reason (broken connection or wrong transfer encoding parameters), the server may lose the exception and receive an empty request body and continue working.
CallLoggingTest is flaky
Test CallLoggingTest.can log application lifecycle events
crashes with the following message:
java.lang.IndexOutOfBoundsException: Index 14 out of bounds for length 6
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at io.ktor.tests.server.features.CallLoggingTest.can log application lifecycle events(CallLoggingTest.kt:64)
Reserved characters in path is not encoded
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1293
Ktor client version: 1.2.3
URLBuilder(host = "localhost").apply {
path("?")
parameters["?"] = "?"
}
.buildString()
.also(::println)
will produce:
http://localhost/??%3F=%3F
I would expect the following url:
http://localhost/%3F?%3F=%3F
URLBuilder
uses encodeURLQueryComponent
function to encode a path: https://github.com/ktorio/ktor/blob/1.2.3/ktor-http/common/src/io/ktor/http/URLBuilder.kt#L50
Possibly it should use encodeURLPath
?
Not sure it is a bug, but a feature. Fixing it will break backward compatibility.
Autoreload embeddedServer configuration fails, if a module has default parameters
Ktor Version
1.4.2
Ktor Engine
Netty
JVM Version, Operating System and Relevant Context
JDK 11, Manjaro
Problem
- Project is generated by Ktor IDEA plugin, which if test modules are included, generates a module with default parameters.
- It fails with Module function provided as lambda cannot be unlinked for reload, if module has default parameters.
- Using method reference as in the example, does not make it work.
Example:
fun main(args: Array<String>): Unit {
embeddedServer(
Netty,
port = 8080,
host = "0.0.0.0",
watchPaths = listOf("some"),
module = Application::module,
).start(true)
}
fun Application.module(testing: Boolean = false): Unit {
routing {
get("/") {
call.respondText { "Hello" }
}
}
}
Running Result
Exception in thread "main" java.lang.RuntimeException: Module function provided as lambda cannot be unlinked for reload
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.<init>(ApplicationEngineEnvironmentReloading.kt:58)
at io.ktor.server.engine.ApplicationEngineEnvironmentBuilder.build(ApplicationEngineEnvironment.kt:105)
at io.ktor.server.engine.ApplicationEngineEnvironmentKt.applicationEngineEnvironment(ApplicationEngineEnvironment.kt:44)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:55)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:36)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:34)
at some.ReloadKt.main(reload.kt:10)
It fails here, basically can't cast the default parameter method as KFunction<*>
public class ApplicationEngineEnvironmentReloading(
//...
) : ApplicationEngineEnvironment {
//...
private val moduleFunctionNames: List<String>? = run {
val configModules = config.propertyOrNull("ktor.application.modules")?.getList()
if (watchPatterns.isEmpty()) configModules
else {
val unlinkedModules = modules.map {
// Fails here
val fn = (it as? KFunction<*>)?.javaMethod
?: throw RuntimeException("Module function provided as lambda cannot be unlinked for reload")
val clazz = fn.declaringClass
val name = fn.name
"${clazz.name}.$name"
}
if (configModules == null)
unlinkedModules
else
configModules + unlinkedModules
}
//...
}
Workaround
fun Application.module() = module(false)
fun Application.module(testing: Boolean = false): Unit {...}
Other
Support explicit WebSocket session close
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1946
Subsystem
Client, and server possibly
Is your feature request related to a problem? Please describe.
client.ws(host = "echo.websocket.org") {
send(Frame.Text("Hello World"))
close()
incoming.consumeEach {
println(it)
}
}
Fails with:
Exception in thread "main" java.util.concurrent.CancellationException: ArrayChannel was cancelled
at kotlinx.coroutines.channels.AbstractChannel.cancel(AbstractChannel.kt:632)
at kotlinx.coroutines.channels.ReceiveChannel$DefaultImpls.cancel$default(Channel.kt:311)
at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl$runOutgoingProcessor$1.invokeSuspend(DefaultWebSocketSessionImpl.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)
at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)
at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:306)
at kotlinx.coroutines.CancellableContinuationImpl.completeResume(CancellableContinuationImpl.kt:393)
at kotlinx.coroutines.channels.AbstractChannel$ReceiveHasNext.completeResumeReceive(AbstractChannel.kt:918)
at kotlinx.coroutines.channels.ArrayChannel.offerInternal(ArrayChannel.kt:83)
at kotlinx.coroutines.channels.AbstractSendChannel.send(AbstractChannel.kt:149)
at io.ktor.http.cio.websocket.WebSocketSession$DefaultImpls.send(WebSocketSessionJvm.kt:45)
at io.ktor.http.cio.websocket.DefaultWebSocketSession$DefaultImpls.send(DefaultWebSocketSession.kt)
at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl.send(DefaultWebSocketSessionImpl.kt:27)
at io.ktor.client.features.websocket.DefaultClientWebSocketSession.send(ClientSessions.kt)
at io.ktor.http.cio.websocket.WebSocketSessionKt.close(WebSocketSession.kt:74)
at io.ktor.http.cio.websocket.WebSocketSessionKt.close$default(WebSocketSession.kt:72)
at WebSocketKt$Websockets$1$1.invokeSuspend(WebSocket.kt:15)
at WebSocketKt$Websockets$1$1.invoke(WebSocket.kt)
at io.ktor.client.features.websocket.BuildersKt.webSocket(builders.kt:54)
at io.ktor.client.features.websocket.BuildersKt$webSocket$1.invokeSuspend(builders.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:272)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at WebSocketKt.Websockets(WebSocket.kt:7)
at MainKt.main(main.kt:4)
Prevent double quotes on header params
Some services require header params to be double-quoted. In contrast, we follow RFC that says that parameter can be non quoted if there are no special characters. To use Ktor client with such services, users add quotes to parameters manually. Sometimes it results in double-quotes.
Exception kotlinx.serialization.SerializationException: Class 'ArrayList' is not registered for polymorphic serialization in the scope of 'Collection' in 1.5.0
When upgrading to 1.5.0 from 1.4.3 I'm getting exception kotlinx.serialization.SerializationException: Class 'ArrayList' is not registered for polymorphic serialization in the scope of 'Collection'
This test triggers the problem (works in 1.4.3):
class ArrayListSerializationBug {
@Test fun testListSerialization() = withTestApplication({
install(ContentNegotiation) {
json()
}
routing {
get("/list") {
val response: Collection<String> = listOf("a", "b")
call.respond(response)
}
}
})
{
with(handleRequest(Get, "/list"), fun TestApplicationCall.() {
assertEquals(OK, response.status())
assertEquals("""["a","b"]""", response.content)
})
}
}
Dispatcher is closing earlier than client
It results in unexpected rejections during decompression and logging:
java.util.concurrent.CancellationException: The task was rejected
at kotlinx.coroutines.ExceptionsKt.CancellationException(Exceptions.kt:22)
at kotlinx.coroutines.ExecutorCoroutineDispatcherBase.cancelJobOnRejection(Executors.kt:144)
at kotlinx.coroutines.ExecutorCoroutineDispatcherBase.dispatch(Executors.kt:97)
Post request shows empty body after upgrading v1.3.2
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1764
Ktor Version 1.3.2 - Android
Post request shows empty body
We are sending a post request using Ktor. After upgrading to V1.3.2 to support kotlin 1.3.7, the request body started showing empty and nothing is actually sent in the body. I used the debugger to trace it, I see that it is able to serialize my request object correctly and content type is done correctly with application/json
, however at some point I see in the logs that it is nullified.
We are using JsonFeature and Logging feature and our event object has it's own serilaizer which I was able to see it successfully serialized inside Ktor client
This is from KotlinxSerializer.kt
override fun write(data: Any, contentType: ContentType): OutgoingContent {
@Suppress("UNCHECKED_CAST")
val content = json.stringify(buildSerializer(data) as KSerializer<Any>, data)
return TextContent(content, contentType)
}
It is returning a valid TextContent(content, contentType)
with correct contentType
and a valid json string as content
This is how I install the features
private val httpClient by lazy {
HttpClient {
install(Logging) {
logger = apiLogger
level = LogLevel.ALL
}
install(JsonFeature) {
serializer = KotlinxSerializer()
}
}
}
This is how we do the post request inside the same class
fun fireEvents(event: List<EventRequest>) = blockingGet {
httpClient.post<Unit>(
scheme = SCHEME_HTTPS,
host = baseUrl,
path = EVENT_API_V1,
body = event
) {
header(CONTENT_TYPE, APPLICATION_JSON)
header(API_TOKEN_HEADER, applicationApiToken)
}
}
In previous version I used to see the body correctly. But now I see the log just BODY START
and BODY END
METHOD: HttpMethod(value=POST)
COMMON HEADERS
-> some-key: some-value
-> Accept: application/json
-> Accept-Charset: UTF-8
CONTENT HEADERS
BODY Content-Type: application/json
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
I/AppLogger: RESPONSE: 200 OK
METHOD: HttpMethod(value=POST)
I/AppLogger: FROM: https://something.com/v1/send
COMMON HEADERS
-> Connection: keep-alive
-> Date: Tue, 31 Mar 2020 14:58:04 GMT
-> Server: nginx/1.14.1
-> Transfer-Encoding: chunked
-> X-Android-Received-Millis: 1585666684496
-> X-Android-Response-Source: NETWORK 200
-> X-Android-Selected-Protocol: http/1.1
-> X-Android-Sent-Millis: 1585666684232
I/AppLogger: BODY Content-Type: null
BODY START
I/AppLogger: BODY END
I am not sure if this is normal or not. But can you tell me if I am doing something wrong in my code that I can fix? I checked changelog but I did not see anything that requires my attention in this code snippet I posted.
A single slash gets ignored for defining a route, but 1.5 requires them due to KTOR-372
route("/resource") {
get("/") {
...
}
}
The above code has the same behavior as this:
route("/resource") {
get("") {
...
}
}
The expected behavior should be the same as this:
route("/resource/") {
get("") {
...
}
}
Ktor 1.5 requires explicitly handling trailing slashes or not, which works in general, for example in the last code snippet here. The "/resource/" is properly parsed to create the equivalent route. But if we "split" the last slash into a nested route like in the first code example, the last slash gets ignored and it is instead treated like the second code snippet. Noted that this only affects the nested routes with no additional path after the slash, a nested get("/abc")
still works.
Expected behavior: Trailing slash in a nested route is properly concatenated to the route
Actual behavior: HTTP 404 Not found when calling the route with a trailing slash.
Provide the capability to highlight breaking changes on the Changelog page breaking-change
it would be useful to have the capability to highlight breaking changes somehow. For example, here we can filter breaking changes, see them for specific versions, and even set the version interval: https://supportcenter.devexpress.com/versionhistory
AFAIU, it requires adding the 'isBreakingChange' field to Yourtrack (or a dedicated tag)
Release 1.5.1
Connect request sends wrong status line
1.5.0
released 22nd December 2020
Client
Ability to send cookies with HttpRequestBuilder
Make curl engine option sslVerify public
Provide/publish internal option to skip TLS verification for curl client engine
Support Java HTTP Client
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1930
Subsystem
Client, JVM
Describe the solution you'd like
In Java 11 was introduced HTTP Client based on reactive-streams.
Motivation to include to ktor
Since more projects are switching to modern Java versions it could be valuable to use built-in Java HTTP Client without need to use additional libraries. It supports HTTP/1.1 and HTTP/2 protocol and provides async APIs.
Apache client engine sometimes hits an unrecoverable socket timeout when using ChannelWriterContent
Test to reproduce below. Tested on two different machines (Windows and macOS), works fine with CIO
. With 500 test iterations, happens at least once most times.
import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.apache.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.http.*
import io.ktor.http.content.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.cio.*
import io.ktor.server.engine.*
import io.ktor.utils.io.*
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.RepeatedTest
import java.util.concurrent.TimeUnit
class KtorApacheBugTest {
private val port = 8000
private val server = embeddedServer(
factory = CIO,
port = port,
) {
routing {
put("/") {
call.receiveChannel().discard()
call.respond(HttpStatusCode.OK)
}
}
}
private val client = HttpClient(Apache) {
install(HttpTimeout) {
socketTimeoutMillis = 10_000
connectTimeoutMillis = 10_000
}
}
@BeforeEach
fun setUp() {
server.start()
}
@AfterEach
fun tearDown() {
client.close()
server.stop(0, 1, TimeUnit.SECONDS)
}
@RepeatedTest(500)
fun `eventually hits a socket timeout`(): Unit = runBlocking {
val chunk = (0 until 10_000).map { it.toByte() }.toByteArray()
client.put<String>("http://localhost:${port}") {
body = ChannelWriterContent(
body = {
writeFully(chunk)
},
contentType = null,
)
}
}
}
Another simple test (seems the issue is with ChannelWriterContent
) below.
This works:
@RepeatedTest(500)
fun `does not hit a socket timeout`(): Unit = runBlocking {
client.put<String>("http://localhost:${port}") {
body = LocalFileContent(File("gradle/wrapper/gradle-wrapper.jar"))
}
}
This does not:
@RepeatedTest(500)
fun `eventually hits a socket timeout`(): Unit = runBlocking {
val chunk = File("gradle/wrapper/gradle-wrapper.jar").readBytes()
client.put<String>("http://localhost:${port}") {
body = ChannelWriterContent(
body = {
writeFully(chunk)
},
contentType = null,
)
}
}
Sending the same via ByteArrayContent
works fine as well.
Allow OkHttpConfig to configure WebSocket.Factory
Currently, Ktor calls OkHttpClient#newWebSocket to create an OkHttp WebSocket:
https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L53
I would like to be able to provide my own instance of OkHttp WebSocket.Factory, so that I may override behaviour. Details of my use case are described here:
https://github.com/square/okhttp/issues/6242
How do you feel about this? Thanks.
CIO: client throws ConnectTimeoutException when endpoint.connectRetryAttempts = 0
The following code fails with a ConnectTimeoutException
.
Maybe the property connectRetryAttempts
is mislabeled and actually means the total number of attempts, not the number of retry attempts.
suspend fun main(): Unit = withContext(Dispatchers.Default) {
HttpClient(CIO) {
engine {
endpoint.connectRetryAttempts = 0
}
}.use { client ->
client.get<String>("https://www.google.com")
}
}
Exception in thread "main" io.ktor.network.sockets.ConnectTimeoutException: Connect timeout has been expired [url=https://www.google.com/, connect_timeout=unknown ms]
at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException(HttpTimeout.kt:164)
at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException$default(HttpTimeout.kt:163)
at io.ktor.client.engine.cio.Endpoint.getTimeoutException(Endpoint.kt:208)
at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:200)
at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Add response text to the message of ResponseException and derived exceptions
message
of ResponseException
contain response's text from response.readText()
.Client sometimes doesn't do response validation
When doing get<HttpResponse> to e.g. only inspect the etag header or some other response header, the status code isn't validated. It's only validated when accessing the body. However, we often don't inspect the body, but only the header. This inconsistency has caused bugs on our side because we missed bad responses.
Could you please always validate the response status code, no matter if the body is accessed or not?
CIO may not work correctly with proxy for https requests
Steps:
- privoxy setup:
$ cat ./privoxyconfig
confdir /etc/privoxy
temporary-directory /tmp
logdir /var/log/privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action # Main actions file
actionsfile user.action # User customizations
filterfile default.filter
filterfile user.filter # User customizations
logfile privoxy.log
debug 1 # Log the destination for each request Privoxy let through. See also debug 1024.
debug 1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
debug 4096 # Startup banner and warnings
debug 8192 # Non-fatal errors
listen-address 0.0.0.0:8118
toggle 1
enable-remote-toggle 0
enable-remote-http-toggle 0
enable-edit-actions 0
enforce-blocks 1
buffer-limit 4096
enable-proxy-authentication-forwarding 0
trusted-cgi-referer http://www.example.org/
forwarded-connect-retries 0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
tolerate-pipelining 1
socket-timeout 300
$ cat ./Dockerfile
FROM alpine
RUN apk add --no-cache privoxy=3.0.28-r0
COPY privoxyconfig /etc/privoxy/config
EXPOSE 8118
CMD privoxy --no-daemon /etc/privoxy/config
$ docker build . -t privoxy
$ docker run --rm -p 127.0.0.1:8118:8118 privoxy
- set proxy as described in https://ktor.io/docs/proxy.html with
fun main(args: Array<String>) {
val client = HttpClient(CIO) {
install(UserAgent) {
agent = "Space Unfurl Bot"
}
install(ContentEncoding) {
gzip()
}
engine {
proxy = Proxy(Proxy.Type.HTTP, InetSocketAddress("127.0.0.1", 8118))
}
}
val r = runBlocking(CoroutineName("Get Metadata")) {
client.get<HttpStatement>("https://example.com").execute { call ->
"response code: ${call.status.description}"
}
}
println(r)
}
Request:
[X] https://example.com
CIO hangs for a long time and returns java.lang.IllegalArgumentException: Invalid TLS record type code: 72
[X] http://ya.ru
same result (ya.ru redirects to https
)
[V] http://example.com
returns page content
[V] Replace CIO with Apache - any requests work perfectly fine.
Core
Staging new public API
Problem
Sometimes, we introduce new API but, due to our compatibility guarantees, we can't make it public and forced to introduce it as internal, sometimes with magic to invoke such internals from other modules. Usually, it is intended to make them public at some point in the future, however, because of long minor/major release cycles, it's easy to forget to promote such APIs.
Proposed solution
The solution is to provide an annotation with the following signature
/**
* API marked with this annotation is intended to become public in the future [version].
* Usually it means that the API can't be public at the moment of development due to
* compatibility guarantees restrictions.
*
* Marking a public declaration with this annotation makes no sense
* except for the case when it is also marked with [InternalAPI].
* Please note that the specified [version] and the fact of making something a candidate is not a guarantee,
* so the target version could be changed without any notice or even the promotion could be cancelled at all.
*
* @property version in which the API is planned to be promoted
*/
@InternalAPI // so users shouldn't use this annotation, it's for ktor only
@Retention(AnnotationRetention.SOURCE)
@Target(
AnnotationTarget.CLASS,
AnnotationTarget.ANNOTATION_CLASS,
AnnotationTarget.CONSTRUCTOR,
AnnotationTarget.FUNCTION,
AnnotationTarget.PROPERTY,
AnnotationTarget.PROPERTY_SETTER,
AnnotationTarget.TYPEALIAS
)
public annotation class PublicAPICandidate(val version: String)
Using this annotation, one could mark some internal API, specifying the desired future version. It is recommended to document such API in spite of that it is internal yet.
When a minor/major version starts (or in the middle of development period), one should search for all annotation usages and collect/notify developers who wanted to make something public. At the end of the version development, we should ensure that there are no annotations with this version remaining.
Every promoted API should be properly documented and have adequate tests.
Example
In 1.4.1 we introduce a property
// In 1.4.1
public class SomeConfig {
/** Initial documentation */
@PublicAPICandidate("1.5.0") // the next minor version
internal var usefulThing: String = ""
}
So no breaking changes, just the internal property.
When we develop 1.5.0 we search for all annotation usages and promote it removing annotation.
// In 1.5.0
public class SomeConfig {
/**
* Documentation
*/
public var usefulProperty: String = ""
}
The other good effect is that by doing this we declare our intention to make something public and declare which version is it. So ktor users may see it, and this avoids unnecessary quesions and improves clarity.
Docs
Documentation for kotlinx.serialization wrong and outdated
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/272
I tried the kotlinx serialization example from the docs: https://ktor.io/servers/features/content-negotiation/serialization-converter.html
Upon using the example I get a deprecation warning for the „serialization(...“ Block because it has been replaced by json.
Second the example does not work. In contrast, the gson example works perfectly fine.
Either fix or remove the section, so other people don’t waste their time trying to get kotlinx.serializtion to work.
Prometheus Registry unavailable for Micrometer
This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/224
The documentation for Micrometer metrics hint towards a Prometheus Registry being available for use here - https://github.com/ktorio/ktorio.github.io/blob/3e1f9c9c52ebbed7f429a05aa1c2b41ed69c6e93/servers/features/metrics-micrometer.md#L97
However, it appears that https://github.com/ktorio/ktor/pull/1105 has stalled out. Can you provide advise on if this feature will still be implemented, if the documentation needs to be updated, or something different. I would also like to know how I can help push this imitative along.
metrics prometheus better documentation
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1842
Subsystem
Server - metrics
Is your feature request related to a problem? Please describe.
https://ktor.io/servers/features/metrics-micrometer.html
- I'm new to ktor and trying to have prometheus metrics, I do it in java app servers with simpleclient_servlet very easy
- I read install MicrometerMetrics, PrometheusMeterRegistry -> in doc missing dependency
- There is no doc where the metrics are exposed(URI)
- There is no doc if metrics need auth and which type of auth
- There is no doc should I need a route which expose metrics
Describe the solution you'd like
- More clear docs or working sample
Motivation to include to ktor
Monitoring of app is very important and prometheus is very popular
clients-index documentation page incorrect link
In this section of the doc page https://ktor.io/docs/clients-index.html#requests-responses - there are two hyperlinks; one for sending requests and one for handling responses.
The "how to receive responses" link seems correct in that it links to the response handling section of the client docs https://ktor.io/docs/response.html.
However, the "how to make requests" appears to link to a server side request handling page https://ktor.io/docs/requests.html#cookies instead of https://ktor.io/docs/request.html#specifying-a-body-for-requests.
The kotlinx.serialization feature documentation is using deprecated API
On page https://ktor.io/docs/serialization-converter.html
Block:
install(ContentNegotiation) {
serialization()
}
has deprecated code.
Apache client engine is documented to support HTTP/2 but doesn't
https://ktor.io/docs/http-client-engines.html#apache
Apache is the most configurable HTTP client about right now. It supports HTTP/1.1 and HTTP/2.
Unfortunately Ktor uses httpclient 4.4 but HTTP/2 support requires httpclient 5.
https://archive.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
https://hc.apache.org/httpcomponents-client-4.4.x/index.html
Public documentation for Ktor 1.4.0 says, that it's in the development
Perhaps I am missing something, but Ktor 1.4.1 is already released.
The same time public documentation for Ktor 1.4.0 says, that it's in development.
As for me, it looks like as it's not yet ready to be used:
❗ You are viewing Ktor 1.4.0 documentation, which is still in development.
Generator
Generate endpoint: 500 server errors without meaningful messages about them
- Server settings endpoint responds with lowercased ID for a build system, but
generate
endpoint expects it to be uppercased and throws an exception:
kotlinx.serialization.SerializationException: io.ktor.generator.web.json.BuildSystemType does not contain element with name 'maven'
- Server settings endpoint responds with lowercased ID for an engine, but
generate
endpoint expects it to be uppercased and throws an exception:
kotlinx.serialization.SerializationException: io.ktor.generator.web.json.Engine does not contain element with name 'netty'
Generate
endpoint responds without a message about absence of dependent feature and just throws an exception:
io.ktor.generator.core.features.MissingRequiredFeatureException: Feature auth depends on the feature routing but it is not selected
Fix height of the right panel on the new generator website.
Currently the panel enhances its size as new features are being added. But I believe the height of the right panel should equal to the max height of the whole window.
Ktor generator produces erroneous build.gradle.kts if KTS is chosen
testImplementation "io.ktor:ktor-server-tests:$ktor_version"
: braces are missingimplementation("io.ktor:ktor-server-core:$ktor_version)")
: odd closing brace
Ktor generator fails to generate when Auth-Oauth is selected
If yo select Oauth feature it will fail to generate project either with a plugin or with a website.
The problem is that it requires Locations feature but it is not aligned with all-features.json
DoubleReceive and Kotlinx.serialization features are missing in generator
Infrastructure
Fix project configuration in IDEA on mingw host
settings.gradle: if (native_targets_enabled) does nothing
The native targets are always include, ignoring the variable. Looks like a wrong merge :)
https://github.com/ktorio/ktor/settings.gradle#L54
if (native_targets_enabled) {
include ':ktor-client:ktor-client-curl'
include ':ktor-client:ktor-client-ios'
}
include ':ktor-client:ktor-client-curl'
include ':ktor-client:ktor-client-ios'
Server
Add option to use specific alias from keystore in CIO TLSConfigBuilder JVM
Subsystem
CIO - JVM
Motivation
This fix adds the option to use only a specific alias instead all alias for a given JVM Keystore.
This change is addictive only, no breaking change.
Solution
Added null-able default null alias which will be use if present. Otherwise, all aliases from a Keystore will be used as known behavior.
https://github.com/ktorio/ktor/pull/2230
Serializing collections of different element types
Original question from https://stackoverflow.com/questions/64319112/why-serializing-collections-of-different-element-types-is-not-supported-in-ktor
To reproduce use the following code example:
@Serializable
sealed class Message {
abstract val content: String
}
@Serializable
data class BroadcastMessage(override val content: String) : Message()
@Serializable
data class DirectMessage(override val content: String, val recipient: String) : Message()
val data: List<Message> = listOf(
DirectMessage("Hey, Joe!", "Joe"),
BroadcastMessage("Hey, all!")
)
fun main() {
embeddedServer(Netty, port = 8080, host = "127.0.0.1") {
install(ContentNegotiation) {
json()
}
routing {
get("/") {
call.respond(data)
}
}
}.start(wait = true)
}
Result:
java.lang.IllegalStateException: Serializing collections of different element types is not yet supported. Selected serializers: [DirectMessage, BroadcastMessage]
at io.ktor.serialization.SerializerLookupKt.elementSerializer(SerializerLookup.kt:71)
at io.ktor.serialization.SerializerLookupKt.serializerForSending(SerializerLookup.kt:41)
at io.ktor.serialization.SerializationConverter.convertForSend(SerializationConverter.kt:114)
at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:149)
at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at ServerKt$main$1$2$1.invokeSuspend(server.kt:39)
at ServerKt$main$1$2$1.invoke(server.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.routing.Routing.executeResult(Routing.kt:148)
at io.ktor.routing.Routing.interceptor(Routing.kt:35)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Support Sealed Classes inside Session-Objects
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/129
ktor not closing user connections
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1885
Ktor Version and Engine Used (client or server and name)
1.3.2, Netty server, routes
ktor_version=1.3.2, kotlin_version=1.3.70
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version")
implementation("io.ktor:ktor-server-netty:$ktor_version")
implementation("ch.qos.logback:logback-classic:$logback_version")
testImplementation("io.ktor:ktor-server-tests:$ktor_version")
Describe the bug
I migrated one of my project form php to ktor and deployed it in production for about 5k users. After one day server stopped working and linux file opened limits exceed which was 20k and when i checked with command ss -ant | grep :8383 | wc -l
connections were 20k+ but i don't have users more than 5k which were not all active. Then i switched my floating ip to old php server and no more request to this server next day when i checked again with command ss -ant | grep :8383 | wc -l
connections were still 20k+ even no more new request were coming and all connections were in ESTAB state.
So after that i created new project which was empty with ktor, kotlin dsl i mentioned its dependencies above and run that on that server Then i run this command watch -n 1 'ss -ant | grep {MY_IP}'
on server to filter connections for my ip first it was no connection from my ip on port 8383 then i send single request from postman on SERVER_IP:8388
which was simple get request and response was from call.respond("Testing")
and on server terminal it show me connection on 8383 with my ip. In postman request was complete and i got result. Then i closed postman and keep checking when it will close my connection from server. I checked that after an hour still connection was not closed and when i requested again it was still using that same connection. So it was reusing connection and not closing even after some hours. I'm not sure but may be error with keep alive or http2. I'm not sending any header i request.
I didn't tried any other ktor or kotlin version and I'm not able to reproduce it local.
I'm running jar file generated with shadowjar on linux server with systemd service.
To Reproduce
- create new project with ktor version 1.3.2 and kotlin 1.3.70 with GradleKotlinDsl
fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
install(Routing) {
get("/") {
call.respond("Testing")
}
}
}
2. create jar file with shadowjar
tasks {
named<ShadowJar>("shadowJar") {
archiveBaseName.set("shadow")
mergeServiceFiles()
manifest {
attributes(mapOf("Main-Class" to mainClasses))
}
transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer::class.java) {
setPath("META-INF/services")
include("org.eclipse.jetty.http.HttpFieldPreEncoder")
}
archiveFileName.set("example.jar")
}
}
- upload on ubuntu server and create its systemd service and run with systemctl start example
service file:
[Unit]
Description=example
After=syslog.target
[Service]
User=root
LimitNOFILE=20536
ExecStart=/var/www/example/example-script
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
example-script:
#!/bin/sh
cd /var/www/example
java -jar exmaple.jar
Expected behavior
When user make request to http://SERVER_IP:APP_PORT it should close it after request complete but it keep opened it until we will stop server. Checking with ss -ant | grep {CLIENT_PUBLIC_IP}
Screenshots
This is screenshot which i get after an hour when made last request in screenshot last connection with port 8383 that is from my machine and it is still not closed.
Jackson: Crash when sending large responses in 1.4.2
Hi there,
We've discovered that Ktor 1.4.2 appears to crash out when sending large responses, using the Jackson feature. I think it might be related to KTOR-1342, but it's also happening on sending. Downgrading to ktor-jackson:1.4.1
seems to fix the issue.
The error logged is:
Exception in thread "eventLoopGroupProxy-3-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@2b5e16e4, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@114e04cb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
I've created a sample project for you to try - https://github.com/SkyeWelch/ktor-1342-error-sample . You can run it with ./gradlew run
, and if it's the same as on my machine see the following responses (I'm using httpie to make requests):
→ http localhost:8080/small
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked
{
"string": "small string"
}
→ http localhost:8080/large
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked
http: error: ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Please let me know if you need anything else.
Thanks,
Skye
GMTDate: Operator overload (plus/minus) for kotlin.time.Duration
As Ktor already has some overloads for kotlin.time.Duration
in ktor-server-core/features/KotlinTimeJvm.kt
, this overload would be nice too.
val date = GMTDate() + 1.hours
/**
* Adds the specified [duration]
*/
@ExperimentalTime
public operator fun GMTDate.plus(duration: Duration): GMTDate = GMTDate(timestamp + duration.toLongMilliseconds())
/**
* Subtracts the specified [duration]
*/
@ExperimentalTime
public operator fun GMTDate.minus(duration: Duration): GMTDate = GMTDate(timestamp - duration.toLongMilliseconds())
Micrometer metrics: scan of random URLs of a server exhausts available values per tag in influx
of("route", call.attributes[measureKey].route ?: call.request.path()),
This line here (https://github.com/ktorio/ktor/blob/master/ktor-features/ktor-metrics-micrometer/jvm/src/io/ktor/metrics/micrometer/MicrometerMetrics.kt#L125) should not assign arbitrary value to a tag.
The result of such behaviour is that if the application is maliciously scanned for every kind of URL possible, the application itself correctly returns 404 on such scans, but the metrics library still reports latency on every single one of the scans with its own tag value. This is very hard, for example, for influx, which has default limit on values per tag (https://stackoverflow.com/questions/43770354/max-values-per-tag-limit-exceeded-influxdb).
And overall, storage of such information is useless.
So there should be some sane limit on possible values of route
tag, possible limited by routes available in application.
Allow any header in CORS configuration
Motivation from the PR: https://github.com/ktorio/ktor/pull/2034
I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load.
I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs
https://expressjs.com/en/resources/middleware/cors.html
"allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex:Content-Type,Authorization
) or an array (ex:['Content-Type', 'Authorization']
). If not specified, defaults to reflecting the headers specified in the request’sAccess-Control-Request-Headers
header."
Get client certificate information from request
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1767
Subsystem
Server
Is your feature request related to a problem? Please describe.
We want to use client-side certificates to authenticate HTTPS requests from connected devices. In our code, we need to get the client certificate information to determine the DN in the certificate.
In a regular servlet, client certificate information can be accessed using request.getAttribute("javax.servlet.request.X509Certificate")
, which returns the certificate chain.
Our strategy to get this to work in ktor was to run our application as a WAR in a Tomcat instance configured to do client authentication. Unfortunately, request attributes do not seem to be available. (Note that a Spring Boot application deployed under the same Tomcat can access these attributes, so we know Tomcat is configured correctly for TLS.)
Describe the solution you'd like
We'd like some way to access servlet request attributes from ktor application code.
Motivation to include to ktor
Client certificates are an important part of IoT deployments, where ktor would shine because typically a server needs to manage a lot of concurrent connections. Ktor itself doesn't support client certificate authentication - our solution would be a workaround if only we could access request attributes.
Implement proper unhandled exception handling strategy
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/82
If exception happens in ktor and is not handled by an interceptor such as StatusPages, it should propagate up to host and be handled there without using any ktor facilities like response pipeline. The reason is that we don't know which part of the system were at fault and it could very well be response pipeline itself. So unhandled exception should be propagated to the host-specific facilities and responded with 500 status code natively there.
Note, that it would make 500 Internal Server Error
status produced in this case not interceptable by StatusPage
feature on a per code basis, but exception filter in the same feature can be installed on Throwable
and do pretty much any custom handling there. At this moment exception would be considered handled and won't be part of this issue.
Support for precompresed files
From PR #540
Nowadays, it is quite common to pre-compress static files ( a.k.a assets ) to reduce response time (no need for on the fly compression).
This pull request enables that, serving the pre-compressed version of the file, suppressing compression(if installed) and responding appropriate Content-Type and Content-Encoding. As brotli is not supported at the time of writing, it also covers serving brotli-compressed files.
Add excludeSuffix to HttpsRedirect feature
Support anyHeader() in CORS feature configuration
Pull request here: https://github.com/ktorio/ktor/pull/2034
I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load.
I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs
https://expressjs.com/en/resources/middleware/cors.html
"allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex: ‘Content-Type,Authorization’) or an array (ex: ['Content-Type', 'Authorization']). If not specified, defaults to reflecting the headers specified in the request’s Access-Control-Request-Headers header."
CORS denies simple request where the content type includes a character encoding
My understanding is that for a CORS simple request to run without prefight, part of the necessary criteria is that it can only have one of three ContentType
values: [application/x-www-form-urlencoded, multipart/form-data, text/plain]
.
Issue KTOR-358 correctly added a check to ensure this criteria is met. But the current implementation doesn't consider when character encoding is appended to ContentType. For example ContentType: text/plain;charset=UTF-8
. If the ContentType is specified in this manner, as is common/default on latest Chrome/Firefox, then the CORS feature will reject it, even though it is a simple request.
As far as I can tell from w3c there is no check required against the charset, rather the specification only cares about the mimeType. However, on reading mdn it says that the "only allowed values" are the three listed above which could be interpreted as not even charset is allowed.
As appending charset is the default on Chrome/Firefox, I think ktor should ignore parameters when parsing ContentType otherwise it will lead to devs setting allowNonSimpleContentTypes = true
which isn't great.
The current CORS.kt feature rejects these requests at line 102:
public suspend fun intercept(context: PipelineContext<Unit, ApplicationCall>) {
// ...
if (!allowNonSimpleContentTypes) {
val contentType = call.request.header(HttpHeaders.ContentType)?.let { ContentType.parse(it) }
if (contentType != null) {
if (contentType !in Configuration.CorsSimpleContentTypes) { // <--- issue
context.respondCorsFailed()
return
}
}
}
//...
}
Configuration.CorsSimpleContentTypes has no values inside parameters, but the parsed value for contentType will contain charset=UTF-8
. And so to fix this, then I think the check should be:
if(contentType.withoutParameters() !in Configuration.CorsSimpleContentTypes)
Thanks!
Deprecate HTTP/2 push support
Since HTTP/2 push will be deprecated so we need to deprecate the corresponding API
Parsing Authorization header throws IllegalArgumentException instead of returning null
Method io.ktor.http.auth.parseAuthorizationHeader(headerValue: String): HttpAuthHeader?
throws IllegalArgumentException instead of returning null in some cases. Looking at the source code, the problem occurs because validation that the method does is weaker than the validation HttpAuthHeader.Single
and HttpAuthHeader.Parametrized
do in their constructors. In effect, when header has non-token68 characters in schema, the method throws when instantiating HttpAuthHeader
.
To reproduce the issue, call the method with value, which includes non-token68 characters before first whitespace:
parseAuthorizationHeader("B<arer MyToken")
To fix the issue, the pattern checked for authScheme should discard values, which don't have token68 characters.
Test Infrastructure
MockEngine regression: request and response history don't support iterator anymore
To reproduce run the following code:
val client = HttpClient(MockEngine) {
engine {
addHandler {
respond("Test")
}
}
}
val engine = client.engine as MockEngine
engine.requestHistory
.map(HttpRequestData::url)
.map(Url::toString)
.count()
I expect no exception, but the java.lang.IllegalStateException
is thrown with Ktor 1.4.0
:
Exception in thread "main" java.lang.IllegalStateException: Common concurrent list doesn't support iterator.
at io.ktor.util.collections.ConcurrentList.iterator(ConcurrentList.kt:101)
at MainKt.main(main.kt:26)
at MainKt.main(main.kt)
No exception is thrown with Ktor 1.3.2-1.4.0-rc
.
Other
URLBuilder to append paths to currently encoded path
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1826
Subsystem
URLBuilder
Is your feature request related to a problem? Please describe.
I want to create a final URL from several paths. Let's say my starting url is https://example.com/items/
and incoming paths are: /category/
, /directory/
, /product
.
Doing:
val url = URLBuilder("https://example.com/items/").run {
path("/category/", "/directory/", "/product")
buildString()
}
I will end up with: https://example.com//category///directory///product
.
Notice that /items/
from starting URL is gone and the unnecessary slash duplications.
To preserve the original path, one can use:
path(encodedPath, "/category/", "/directory/", "/product")
but that does not fix the path separator problem (in case where you don't control them).
Describe the solution you'd like
Adding a simple function, appendToPath
, which will handle path extending and separators.
Exemplary implementation:
fun URLBuilder.appendToPath(vararg components: String) {
// removes all trailing and leading separators
val paths = components
.map { it.replace(Regex("""(^/|/+$)"""), "") }
.joinToString("/", transform = { it.encodeURLQueryComponent() })
// make sure that there's a slash separator at the end of current path
if (!encodedPath.endsWith('/')) {
encodedPath = "${encodedPath}/"
}
this.encodedPath += paths
}
Motivation to include to ktor
I think this might be a common case (building up URL), so in my opinion, this small helper would be a beneficial addition. I'm more than happy to open up PR.
Code autoreload not working for me
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/975
Ktor Version
1.1.3
Ktor Engine Used(client or server and name)
EngineMain
I guess?
JVM Version, Operating System and Relevant Context
JDK 1.8. Windows 10
Feedback
Okay I have a new project generated using the Ktor IDEA plugin. I'm just trying to make autoreload work (which should by default?). I have only one Kotlin file in src/Application.kt
. My application.conf
looks like this:
ktor {
deployment {
port = 8080
port = ${?PORT}
autoreload = true
watch = [ MyProjectFolder ]
}
application {
modules = [ com.company.ApplicationKt.module ]
}
}
The issue is, when I change Application.kt
the project doesn't reload and the changes are not reflected in the browser.
I see this in the run output:
2019-02-25 11:01:46.894 [main] DEBUG Application - Watching F:\Projects\MyProjectFolder\out\production\classes for changes.
2019-02-25 11:01:46.895 [main] DEBUG Application - Watching F:\Projects\MyProjectFolder\out\production\classes\com for changes.
...
I also tried com.company
instead of MyProjectFolder
in the watch list but that one is even worse. The run output shows autoreload is disabled.
Ktor should resolve trailing slash as a different URL
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1876
From the beginning to Ktor 1.3.2.
Ktor makes no difference between URLs ending with a trailing slash or not.
The following code will produce an error because two handlers are trying to respond to the incoming request.
@Test fun `routing trailing slash`() = withTestApplication {
application.routing {
get("foo") {
println("server processing /foo")
call.respondText("/foo")
}
get("foo/") {
println("server procession /foo/")
call.respondText("/foo/")
}
}
on ("making /foo request") {
val result = handleRequest {
uri = "/foo"
method = HttpMethod.Get
}
it("should be handled") {
assertTrue(result.requestHandled)
}
it("should have /foo as a response") {
assertEquals("/foo", result.response.content)
}
}
}
//produces
server processing /foo
server procession /foo/
Response has already been sent
io.ktor.server.engine.BaseApplicationResponse$ResponseAlreadySentException: Response has already been sent
This error has an impact on SEO because Google respects the difference between /foo
and /foo/
that may produce different results.
It is also a problem when you want to use a first-level path parameter like "/{user}/"
coupled with static content serving. Browsers can send requests to get a favicon file. These requests don't have any trailing slash, but still, Ktor will route them to the "/{user}/"
handler instead of serving the file.
Expected behavior
A router "/{user}/" should not handle a "/favicon.png" request.
Because it is a modification of behavior, users should activate this correction by configuration.
Implement Support for Compression Extensions for WebSocket (RFC 7692)
Upgrade gradle to 6.7.1 and enable fs watcher
Fix failing tests in master after 1.5.0 merge
Implement development mode for Ktor
Fix typo `val socketTimeout` in `CIOEngineConfig` cause it's a property in the config
Content-Disposition additional parameters should be inside quotes
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1691
Ktor Version and Engine Used (client or server and name)
1.3.1, client
Describe the bug
Content-Disposition
additional parameters should be inside quotes in multi-part Request body.
To Reproduce
Steps to reproduce the behavior:
- Multi-part request
import io.ktor.client.request.forms.append
<...>
httpClient.post<String> {
body = MultiPartFormDataContent(formData {
append(
"file",
"file.txt",
ContentType.parse("text/plain")
) {
writeText("content")
}
})
}
<...>
- Observe part data in Request body:
Content-Disposition: form-data; name=file; filename=file.txt
Expected behavior
Header matching spec:
The first parameter in the HTTP context is always form-data. Additional parameters are case-insensitive and have arguments that use quoted-string syntax after the '=' sign. Multiple parameters are separated by a semi-colon (';').
Content-Disposition: form-data; name="file"; filename="file.txt"
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
OAuth feature is incompatible with Dropbox
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/771
Ktor Version
1.0.0
Ktor Engine Used(client or server and name)
Server: Netty
Client: Apache
Using OAuth feature
JVM Version, Operating System and Relevant Context
JVM: 10
OS: Mac
Feedback
OAuth2 with Dropbox fails during the "request access token" step. In summary, the OAuth feature includes the state
parameter in the request. Dropbox's API does not allow unexpected parameters, and fails the OAuth access token request.
From what I can tell, there is currently no way to prevent the state from being added to the access token request.
Details
Consider the following 3 client-server exchanges. Here the Client is the Ktor server, and the Server is the identity provider (Dropbox, Google, Github, etc).
- [Client] --
/authorize
-> [Server] - [Client] <-
...?code=...
-- [Server] - [Client] --
/access_token
-> [Server]
In a typical OAuth2 exchange, the Client sends a state=<nonce>
parameter with the request in 1
and the Server responds with state=<nonce>
in the redirect in 2
. (This state sending is not mandatory).
The OAuth feature is configured to include the state
in 3
when the Client requests the access token from the Server. Some APIs such as Github's want the state to be included in the request. Dropbox however, reject requests that include the state.
The following code sets the state
on the access token request.
https://github.com/ktorio/ktor/blob/85210082c729b12099ad25f509e99b23ccea23b8/ktor-features/ktor-auth/src/io/ktor/auth/OAuth2.kt#L149-L151
In short, if a state
was returned from the Server in 2
, it will be non-null
at this point in the code.
Using a custom OAuth2StateProvider
OAuth2ServerSettings
allows the settings of a custom OAuth2StateProvider
with which we can generate a custom state
. I mentioned previously that the state
is not mandatory, so maybe we can not generate a state? This is not an option, as the state
must not be null
. An empty state
results in a request that has state=
, but state is still there.
java.lang.IllegalStateException: Reading is not available in state Reading
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/813
Ktor Version
1.0.1
Ktor Engine Used(client or server and name)
Netty / Firefox
JVM Version, Operating System and Relevant Context
1.8 Windows Server 2008R2
Feedback
I don't how this happen, I can't reproduce it.
stacktrace:
500 Internal Server Error: POST - /search/TT
java.lang.IllegalStateException: Reading is not available in state Reading
at kotlinx.coroutines.io.internal.ReadWriteBufferState.startReading$kotlinx_coroutines_io_jvm(ReadWriteBufferSta
te.kt:19)
at kotlinx.coroutines.io.ByteBufferChannel.setupStateForRead(ByteBufferChannel.kt:238)
at kotlinx.coroutines.io.ByteBufferChannel.access$setupStateForRead(ByteBufferChannel.kt:21)
at kotlinx.coroutines.io.ByteBufferChannel.lookAhead(ByteBufferChannel.kt:2868)
at kotlinx.coroutines.io.DelimitedKt.readUntilDelimiter(Delimited.kt:24)
at io.ktor.http.cio.MultipartKt.copyUntilBoundary(Multipart.kt:339)
at io.ktor.http.cio.MultipartKt$copyUntilBoundary$1.invokeSuspend(Multipart.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(Dispatched.kt:111)
at kotlinx.coroutines.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:
83)
at kotlinx.coroutines.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2207)
at kotlinx.coroutines.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:865)
at kotlinx.coroutines.io.ByteBufferChannel.readAsMuchAsPossible(ByteBufferChannel.kt:426)
at kotlinx.coroutines.io.ByteBufferChannel.readAvailable(ByteBufferChannel.kt:559)
at kotlinx.coroutines.io.ByteBufferChannel.readAvailableSuspend(ByteBufferChannel.kt:593)
at kotlinx.coroutines.io.ByteBufferChannel$readAvailableSuspend$2.invokeSuspend(ByteBufferChannel.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:236)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Improve changelog of ktor.io
- Add release date for each release
- Add the link to the changelog on the main page