There are cases where the client of a digest authentication has no knowledge of the realm value. For this scenario, a case in which the realm contained in the 401 response is immediately used should be added.

Reproducer:

suspend fun main() {
    val channel = ByteChannel(true)

    println(channel.availableForRead) //prints 0
    channel.writePacket {
        writeText("hello")
    }
    println(channel.availableForRead) //prints 5
    val packet = channel.readRemaining(channel.availableForRead.toLong()) //suspend...
//    val packet = channel.readPacket(channel.availableForRead)
    println(packet.readText())
}

Channel has 5 bytes available for read. Contract of availableForRead says, that:

    /**
     * Returns number of bytes that can be read without suspension. Read operations do no suspend and return
     * immediately when this number is at least the number of bytes requested for read.
     */
    public actual val availableForRead: Int

But in readRemaining it suspends on trying to read... Using readPacket it works ok.

Using ByteChannelSequential on JVM don't suspend on both calls. On K/N the same - both don't suspends.

Looks like it regression after KTOR-1268. Caused by this change: https://github.com/ktorio/ktor/pull/2214/files#diff-408be3eaf4a8ec196449c0483c1525a24450590d9473ebcc3a5bc3798326e3e7L2234-R2222 Now channel is trying to read one byte, even if it's not needed.

Reproduced on: kotlin: 1.4.32 ktor: 1.5.3

Minimal reproducer: on K/N:

fun main() = runBlocking {
    val channel = ByteChannel(true)
    channel.writePacket {
        writeText("h")
    }
    channel.close()
    val packet = withTimeoutOrNull(1000) {
        //    val packet = channel.readRemaining(5L)
        channel.readPacket(5)
    }
    println(packet?.readText())
}

Channel closed, but readPacket freezes forever - even not fail by timeout. Same behavior is if using ByteChannelSequentialJVM(IoBuffer.Empty, true) on K/JVM. If using ByteBufferChannel on JVM - behavior is ok, exception is thrown:

Exception in thread "main" kotlinx.coroutines.channels.ClosedReceiveChannelException: Unexpected EOF: expected 4 more bytes
	at io.ktor.utils.io.ByteBufferChannel.readFullySuspend(ByteBufferChannel.kt:598)
	at io.ktor.utils.io.ByteBufferChannel.readFully(ByteBufferChannel.kt:590)
	at io.ktor.utils.io.ByteBufferChannel.readPacketSuspend(ByteBufferChannel.kt:806)
	at io.ktor.utils.io.ByteBufferChannel.readPacket$suspendImpl(ByteBufferChannel.kt:792)
	at io.ktor.utils.io.ByteBufferChannel.readPacket(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteReadChannelKt.readPacket(ByteReadChannel.kt:202)
	at AppKt$main$1$packet$1.invokeSuspend(App.kt:57)

If to use readRemaining - for any channel, behavior is the same, it reads packet with one letter and prints it.

@OptIn(InternalAPI::class)
suspend fun main() {
    val connection = aSocket(SelectorManager()).tcp().bind(port = 8000).accept().connection()
    while (true) {
        delay(1000)
        connection.output.writePacket(buildPacket {
            writeText("1".repeat(128))
        })
        connection.output.flush()
    }
}

@OptIn(InternalAPI::class)
fun main() = runBlocking {
    val connection = aSocket(SelectorManager()).tcp().connect("0.0.0.0", port = 8000).connection()
    while (true) {
        val packet = withTimeoutOrNull(5000) {
            println("await")
            val packet = connection.input.readRemaining(128)
//            val packet = connection.input.readPacket(128)
            require(packet.remaining == 128L) { "Length should be 128 but was ${packet.remaining}" }
            packet
        }
        println("received: ${packet?.readText()}")
    }
}

await
received: 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
await
received: 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

Uncaught Kotlin exception: kotlin.IllegalArgumentException: Length should be 128 but was 0

Exception in thread "main" java.lang.IllegalArgumentException: Length should be 128 but was 0

Exception in thread "main" kotlinx.coroutines.channels.ClosedReceiveChannelException: Unexpected EOF: expected 128 more bytes
	at io.ktor.utils.io.ByteBufferChannel.readFullySuspend(ByteBufferChannel.kt:598)
	at io.ktor.utils.io.ByteBufferChannel$readFullySuspend$1.invokeSuspend(ByteBufferChannel.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:86)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:61)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at AppKt.main(App.kt:24)
	at AppKt.main(App.kt)

Reproduced on: kotlin: 1.4.32 ktor: 1.5.3 os: macos K/N target: macos, both release and debug

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/541

Simple demo:

https://github.com/ktorio/ktor-samples/compare/master...marshallpierce:html-builder-status-pages-swallows-exceptions?expand=1

Expected behavior: I would expect the default to involve a 500 and the exception being logged somewhere, hopefully customizable via StatusPages. Rendering part of the DSL's execution with a 200 is highly misleading, to say the least.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/96

Pages like https://ktor.io/quickstart/quickstart/gradle.html lack ready-to-download (and use) example projects

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1004

Ktor Version

1.1.3 with Tomcat and Gradle

Operating System

Operating system: MacOS High Sierra

Feedback

By following the steps in documentation to generate a FatJar, whenever I try to execute my Ktor application by using the command java -jar pathToFatJar, I receive de following error:

Exception in thread "main" java.lang.UnsupportedOperationException: Packages and file facades are not yet supported in Kotlin reflection. Meanwhile please use Java reflection to inspect this class: class com.tidelevel.ApplicationKt
	at kotlin.reflect.jvm.internal.KClassImpl.reportUnresolvedClass(KClassImpl.kt:301)
	at kotlin.reflect.jvm.internal.KClassImpl.access$reportUnresolvedClass(KClassImpl.kt:43)
	at kotlin.reflect.jvm.internal.KClassImpl$Data$descriptor$2.invoke(KClassImpl.kt:53)
	at kotlin.reflect.jvm.internal.KClassImpl$Data$descriptor$2.invoke(KClassImpl.kt:44)
	at kotlin.reflect.jvm.internal.ReflectProperties$LazySoftVal.invoke(ReflectProperties.java:92)
	at kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:31)
	at kotlin.reflect.jvm.internal.KClassImpl$Data.getDescriptor(KClassImpl.kt)
	at kotlin.reflect.jvm.internal.KClassImpl$Data$objectInstance$2.invoke(KClassImpl.kt:106)
	at kotlin.reflect.jvm.internal.ReflectProperties$LazyVal.invoke(ReflectProperties.java:62)
	at kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:31)
	at kotlin.reflect.jvm.internal.KClassImpl$Data.getObjectInstance(KClassImpl.kt)
	at kotlin.reflect.jvm.internal.KClassImpl.getObjectInstance(KClassImpl.kt:239)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createModuleContainer(ApplicationEngineEnvironmentReloading.kt:328)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:317)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:275)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:127)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:247)
	at io.ktor.server.tomcat.TomcatApplicationEngine.start(TomcatApplicationEngine.kt:108)
	at io.ktor.server.tomcat.EngineMain.main(EngineMain.kt:16)

This error just happen when trying to execute the FatJar in command line, everything works fine when running in Intellij. I have seen similar issues at:

• https://github.com/ktorio/ktor/issues/386
• https://stackoverflow.com/questions/54282790/ktor-running-fat-jar-throws-java-lang-unsupportedoperationexceptionpackages-an

By changing the mainClassName property to correspond to my own Application class, everything seems to work fine! I am not sure if the documentation needs to be updated in this page or if I did something wrong.

Hope I receive an answer from you!

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/268

https://ktor.io/clients/http-client/features/timeout.html

The Client Timeout Feature states that "By default all these timeouts are infinite"; this doesn't seem to be the case with either the Apache or CIO engines where unless I explicitly set a timeout I get:

Apache

io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=http://localhost:1080/api/import/tickets/v1/file, socket_timeout=unknown] ms
	at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:170)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt$sendRequest$$inlined$suspendCancellableCoroutine$lambda$2.failed(ApacheHttpRequest.kt:44)
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: 10,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	... 11 common frames omitted

CIO (which doesn't seem to accept specifying timeouts at all)

kotlinx.coroutines.TimeoutCancellationException: Timed out waiting for 15000 ms
	at kotlinx.coroutines.TimeoutKt.TimeoutCancellationException(Timeout.kt:149)
	at kotlinx.coroutines.TimeoutCoroutine.run(Timeout.kt:119)
	at kotlinx.coroutines.EventLoopImplBase$DelayedRunnableTask.run(EventLoop.common.kt:493)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:272)
	at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:68)
	at java.base/java.lang.Thread.run(Thread.java:834)

Should cover multiplatform scenarios also

https://ktor.io/clients/http-client/quick-start/requests.html -- This is another page that describes post requests (among with other requests). First of all, it does not teach what is contentType parameter of the TextContent constructor (please, add explanation) Second of all, the only example about using JsonFeature tells:

If you install the JsonFeature, and set the content type to application/json you can use arbitrary instances as the body, and they will be serialized as JSON:

but in the code snippet provided for this example there is no contentType set, and it has incorrect code (please, fix):

// !!!!!!!    @Serializable is missing here !!!!!!
data class HelloWorld(val hello: String)

val client = HttpClient(Apache) {
    install(JsonFeature) {
        serializer = GsonSerializer {
            // Configurable .GsonBuilder
            serializeNulls()
            disableHtmlEscaping()
        }
    }
}

client.post<Unit> {
    url("http://127.0.0.1:8080/")
    body = HelloWorld(hello = "world")
    // !!!!!!!    contentType(ContentType.Application.Json) is missing here !!!!!!
}

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/386

Followed instruction here https://ktor.io/servers/deploy.html for fat jar deployment

When I run java -jar myjar.jar , i get main class could'nt be found in io.ktor.server.netty.DevelopmentEngine.

Project runs fine in IDEA

Any idea why?

IntelliJ IDEA 2021.1 with Ktor plugin 1.5.3

1. New Project - Ktor
2. Add Feature - Thymeleaf
3. Open Templating.kt file

Actual result: templates/thymeleaf/ is mentioned in code but there is no such directory and index file in project

fun Application.configureTemplating() {
    install(Thymeleaf) {
        setTemplateResolver(ClassLoaderTemplateResolver().apply {
            prefix = "templates/thymeleaf/"
            suffix = ".html"
            characterEncoding = "utf-8"
        })
    }

    routing {
        get("/html-thymeleaf") {
            call.respond(ThymeleafContent("index", mapOf("user" to ThymeleafUser(1, "user1"))))
        }
    }
}

To reproduce create a new project via the IDEA plugin without any feature and run generated application using Gradle:

./gradlew run

As a result the app crashes with the following error:

> Task :run FAILED
2021-04-08 16:20:57.086 [main] TRACE Application - No configuration provided: neither application.conf nor system properties nor command line options (-config or -P:ktor...=) provided
Exception in thread "main" java.lang.IllegalArgumentException: Neither port nor sslPort specified. Use command line options -port/-sslPort or configure connectors in application.conf
	at io.ktor.server.engine.CommandLineKt$commandLineEnvironment$environment$1.invoke(CommandLine.kt:142)
	at io.ktor.server.engine.CommandLineKt$commandLineEnvironment$environment$1.invoke(CommandLine.kt)
	at io.ktor.server.engine.ApplicationEngineEnvironmentBuilder.build(ApplicationEngineEnvironment.kt:110)
	at io.ktor.server.engine.ApplicationEngineEnvironmentKt.applicationEngineEnvironment(ApplicationEngineEnvironment.kt:46)
	at io.ktor.server.engine.CommandLineKt.commandLineEnvironment(CommandLine.kt:61)
	at io.ktor.server.netty.EngineMain.main(EngineMain.kt:21)

Adding "Gson" and "Kotlinx.Serialization" dependencies in the same wizard project generates the following code:

install(ContentNegotiation) {
    gson {
    }
}
install(ContentNegotiation) {
    json()
}

// . . .

routing {
    get("/json/gson") {
        call.respond(mapOf("hello" to "world"))
    }
}
routing {
    get("/json/kotlinx-serialization") {
        call.respond(mapOf("hello" to "world"))
    }
}

This (expectedly) crashes at first run:

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)

Caused by: io.ktor.application.DuplicateApplicationFeatureException: Conflicting application feature is already installed with the same key as `ContentNegotiation`
	at io.ktor.application.ApplicationFeatureKt.install(ApplicationFeature.kt:82)
	at com.example.ApplicationKt.module(Application.kt:27)
	at com.example.ApplicationKt.module$default(Application.kt:21)

If you want to build KTOR on a new fresh Mac, you need curl. The primary option to install curl is using Homebrew, which change the default installation path from /usr/local to /opt/homebrew from 2.5.10. The installation path will only change for new brew installation, to prevent crashing stable installation. On Apple M1 chip, it will always use the new installation path by default.

Fix: Add "/opt/homebrew/opt/curl/include/curl", to the build script

When selecting the CIO option in the Project Wizard, it uses "Netty" as parameter to the embeddedServer function as opposed to CIO.

IJ 212 Nightly

1. Open New Project wizard
2. Select Ktor
3. Close New Project wizard

2021-04-30 10:44:27,515 [ 181985]  ERROR - tellij.openapi.util.ObjectTree - Memory leak detected: 'io.ktor.initializr.intellij.features.KtorFeaturesStep' of class io.ktor.initializr.intellij.features.KtorFeaturesStep is registered in Disposer but wasn't disposed.
Register it with a proper parentDisposable or ensure that it's always disposed by direct Disposer.dispose call.
See https://jetbrains.org/intellij/sdk/docs/basics/disposers.html for more details.
The corresponding Disposer.register() stacktrace is shown as the cause:
 
java.lang.RuntimeException: Memory leak detected: 'io.ktor.initializr.intellij.features.KtorFeaturesStep' of class io.ktor.initializr.intellij.features.KtorFeaturesStep is registered in Disposer but wasn't disposed.
Register it with a proper parentDisposable or ensure that it's always disposed by direct Disposer.dispose call.
See https://jetbrains.org/intellij/sdk/docs/basics/disposers.html for more details.
The corresponding Disposer.register() stacktrace is shown as the cause:

	at com.intellij.openapi.util.ObjectTree.assertIsEmpty(ObjectTree.java:227)
	at com.intellij.openapi.util.Disposer.assertIsEmpty(Disposer.java:167)
	at com.intellij.openapi.util.Disposer.assertIsEmpty(Disposer.java:162)
	at com.intellij.openapi.application.impl.ApplicationImpl.disposeContainer(ApplicationImpl.java:246)
	at com.intellij.openapi.application.impl.ApplicationImpl.disposeSelf(ApplicationImpl.java:263)
	at com.intellij.openapi.application.impl.ApplicationImpl.doExit(ApplicationImpl.java:650)
	at com.intellij.openapi.application.impl.ApplicationImpl.exit(ApplicationImpl.java:603)
	at com.intellij.openapi.application.impl.ApplicationImpl.exit(ApplicationImpl.java:592)
	at com.intellij.openapi.application.ex.ApplicationEx.exit(ApplicationEx.java:73)
	at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeFrame$3.windowClosing(WelcomeFrame.java:117)
	at java.desktop/java.awt.AWTEventMulticaster.windowClosing(AWTEventMulticaster.java:357)
	at java.desktop/java.awt.AWTEventMulticaster.windowClosing(AWTEventMulticaster.java:357)
	at java.desktop/java.awt.Window.processWindowEvent(Window.java:2090)
	at java.desktop/javax.swing.JFrame.processWindowEvent(JFrame.java:298)
	at java.desktop/java.awt.Window.processEvent(Window.java:2049)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5027)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2784)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:896)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:768)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:434)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:825)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:433)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:842)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:487)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)
Caused by: java.lang.Throwable
	at com.intellij.openapi.util.ObjectNode.<init>(ObjectNode.java:31)
	at com.intellij.openapi.util.ObjectTree.createNodeFor(ObjectTree.java:99)
	at com.intellij.openapi.util.ObjectTree.register(ObjectTree.java:60)
	at com.intellij.openapi.util.Disposer.register(Disposer.java:73)
	at com.intellij.util.Alarm.<init>(Alarm.java:130)
	at com.intellij.openapi.ui.LoadingDecorator.<init>(LoadingDecorator.java:54)
	at com.intellij.openapi.ui.LoadingDecorator.<init>(LoadingDecorator.java:47)
	at com.intellij.openapi.ui.LoadingDecorator.<init>(LoadingDecorator.java:43)
	at com.intellij.ui.components.JBLoadingPanel.lambda$new$0(JBLoadingPanel.java:29)
	at com.intellij.ui.components.JBLoadingPanel.<init>(JBLoadingPanel.java:38)
	at com.intellij.ui.components.JBLoadingPanel.<init>(JBLoadingPanel.java:29)
	at io.ktor.initializr.intellij.features.KtorFeaturesStep.<init>(KtorFeaturesStep.kt:33)
	at io.ktor.initializr.intellij.KtorModuleBuilder.createWizardSteps(KtorModuleBuilder.kt:63)
	at com.intellij.ide.util.newProjectWizard.StepSequence.addStepsForBuilder(StepSequence.java:52)
	at com.intellij.ide.projectWizard.ProjectTypeStep.<init>(ProjectTypeStep.java:234)
	at com.intellij.ide.projectWizard.NewProjectWizard.init(NewProjectWizard.java:57)
	at com.intellij.ide.projectWizard.NewProjectWizard.<init>(NewProjectWizard.java:46)
	at com.intellij.ide.actions.NewProjectAction.actionPerformed(NewProjectAction.java:20)
	at com.intellij.openapi.actionSystem.ex.ActionUtil.performAction(ActionUtil.java:247)
	at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAware(ActionUtil.java:236)
	at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAwareWithCallbacks(ActionUtil.java:229)
	at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeScreenActionsUtil.performAnActionForComponent(WelcomeScreenActionsUtil.java:96)
	at com.intellij.openapi.wm.impl.welcomeScreen.WelcomeScreenActionsUtil$ToolbarTextButtonWrapper$1.actionPerformed(WelcomeScreenActionsUtil.java:56)
	at java.desktop/javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967)
	at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308)
	at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405)
	at java.desktop/javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262)
	at java.desktop/javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:270)
	at java.desktop/java.awt.Component.processMouseEvent(Component.java:6652)
	at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3345)
	at java.desktop/java.awt.Component.processEvent(Component.java:6417)
	at java.desktop/java.awt.Container.processEvent(Container.java:2263)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5027)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859)
	at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918)
	at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547)
	at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2784)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751)
	at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:896)
	at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.java:828)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:765)
	... 11 more

To reproduce:

1. Create KMM project
2. Add Ktor module Expected: Ktor module is added Actual: Ktor module is not added with an error message "Error adding module to project: Module already exists"

On the attached video root module is called "karta" and new Ktor module is "zzz", but the error eventually says that module "karta" already exists.

Tried to restart IDE and and invalidate caches.

IU-211.6693.111, JRE 11.0.10+9-b1341.35x64 JetBrains s.r.o., OS Mac OS X(x86_64) v11.2.3, screens 5120.0x2880.0, 5120.0x2880.0, 2880.0x5120.0; Retina

Example:

route("path1/path2") {
    get {}
    post {}
}

path1/path2 will not be visible in Endpoints View while it must be there twice: for POST and for GET methods.

This test passes -- note the ///

    @Test
    internal fun routeSlashes() {
        withTestApplication({
            routing {
                get("/") {
                    call.respond("OK: ${(call as RoutingApplicationCall).route}")
                }
            }
        }) {
            with(handleRequest(HttpMethod.Get, "/")) {
                assertEquals("OK: ///(method:GET)", response.content)
            }
        }
    }

If the route is declared with get(""), Route.toString() results in /(method:GET), which is what I would expect in both cases.

Including Kotlinx HTML DSL as the templating language results in the following line in the configuration:

maven { url = uri("https://maven.pkg.jetbrains.space/public/p/kotlinx-html/maven") }

Since kotlinx-html-jvm:0.7.3 is already published on mavenCentral, there is no need for this line anymore going forward, and it can be removed. (Relates to KTOR-2203, which can be closed, I suppose.)

If you have a common module containing the api for your app, you can't annotate the functions with @Throws(IosHttpRequestException::class), because IosHttpRequestException is defined in the iOS client only and is a subclass from Exception. Adding @Throws(Exception::class) is not a good choice, because a possible wrong request builder is catched too (eg invalide urlString).

class API(val client: HttpClient) {

@Throws(IosHttpRequestException::class) // not possible in commonMain
suspend fun users() = client.get<List<User>>("/users")
}

solution

class API(val client: HttpClient) {

@Throws(IOException::class)
suspend fun users() = client.get<List<User>>("/users")
}

PR: https://github.com/ktorio/ktor/pull/2436

After updating from ktor 1.4.3 to 1.5.2 and running my integration tests using ktor I'm running into:

java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
	at io.ktor.util.Attributes$DefaultImpls.get(Attributes.kt:29)
	at io.ktor.util.AttributesJvmBase.get(AttributesJvm.kt:15)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:28)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
	at io.ktor.client.features.HttpCallValidator.validateResponse(HttpCallValidator.kt:55)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:134)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invoke(HttpCallValidator.kt)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:98)
	at ???(Coroutine boundary.?(?)
	at app.becoach.roboter.CoacheeRoboter$login$1.invokeSuspend(CoacheeRoboter.kt:26)
Caused by: java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
	at io.ktor.util.Attributes$DefaultImpls.get(Attributes.kt:29)
	at io.ktor.util.AttributesJvmBase.get(AttributesJvm.kt:15)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:28)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
	at io.ktor.client.features.HttpCallValidator.validateResponse(HttpCallValidator.kt:55)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:134)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invoke(HttpCallValidator.kt)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:98)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Do you happen to have any idea what's causing this? My unit tests are getting 4xx error codes from the backend. I've got a custom Feature installed where upon unauthorized error codes, I'm re-issuing a new token.

My feature looks like this:

internal class AuthFeature(
  private val credentials: String,
  private val tokenStorage: TokenStorage,
  private val logger: Logger
) {
  class Config {
    lateinit var credentials: String
    lateinit var tokenStorage: TokenStorage
    lateinit var logger: Logger
  }

  companion object Feature : HttpClientFeature<Config, AuthFeature> {
    override val key: AttributeKey<AuthFeature> = AttributeKey("AuthFeature")

    /**
     * The Mutex is required in order to only refresh once when multiple requests
     * are being triggered simultaneously e.g. at app startup.
     */
    private val mutex = Mutex()

    override fun prepare(block: Config.() -> Unit): AuthFeature {
      val config = Config().apply(block)
      return AuthFeature(
          credentials = config.credentials,
          tokenStorage = config.tokenStorage,
          logger = config.logger,
      )
    }

    @KtorExperimentalAPI
    override fun install(feature: AuthFeature, scope: HttpClient) {
      scope.requestPipeline.intercept(HttpRequestPipeline.State) {
        val accessToken = feature.tokenStorage.getAccessToken()
        val isRefreshingToken = context.url.buildString().endsWith("oauth/token")
        val header = if (accessToken != null && !isRefreshingToken) "Bearer $accessToken" else feature.credentials
        context.headers.append(HttpHeaders.Authorization, header)
      }

      requireNotNull(scope.feature(HttpSend)).intercept { call, _ ->
        if (call.response.status == HttpStatusCode.Unauthorized) {
          val mutexId = uuid4().toString()
          val tag = "${key.name}-$mutexId"
          feature.logger.i(tag, "Got ${HttpStatusCode.Unauthorized.value}")

          val beforeMutexAccessToken = feature.tokenStorage.getAccessToken()
          val request = call.request

          mutex.withLock {
            val inMutexAccessToken = feature.tokenStorage.getAccessToken()
            val needsToRefresh = beforeMutexAccessToken == inMutexAccessToken

            if (needsToRefresh) {
              feature.tokenStorage.putAccessToken(null) // Invalidate access token.
            }

            val refreshToken = feature.tokenStorage.getRefreshToken()

            if (refreshToken != null) {
              if (needsToRefresh) {
                feature.logger.i(tag, "Issuing new access token")
                val refreshResponse = scope.refreshToken(refreshToken)
                feature.logger.i(tag, "Updating access & refresh token")

                feature.tokenStorage.putAccessToken(refreshResponse.accessToken)
                feature.tokenStorage.putRefreshToken(refreshResponse.refreshToken)
              } else {
                feature.logger.i(tag, "Other request already refreshed token")
              }

              // Build previous request once again.
              val requestBuilder = HttpRequestBuilder()
              requestBuilder.takeFrom(request)

              // Force the new token by explicitly setting it.
              requestBuilder.headers[HttpHeaders.Authorization] = "Bearer ${feature.tokenStorage.getAccessToken()}"

              feature.logger.i(tag, "Retrying previous request with new access token")
              execute(requestBuilder).also {
                feature.logger.i(tag, "Finished previous request")
              }
            } else {
              feature.logger.i(tag, "No refresh token and hence leaving as is")
              call
            }
          }
        } else {
          call
        }
      }
    }
  }
}

Test history: https://ktor.beta.teamcity.com/test/451494659026389538?currentProjectId=Ktor_ProjectKtorCore&branch=&orderBy=status&order=desc&expandedTest=id%3A23376%2Cbuild%3A%28id%3A24241%29

java.net.SocketException: Connection reset by peer
  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
  at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:38)
  at io.ktor.network.sockets.SocketImpl$connect$1.invokeSuspend(SocketImpl.kt)
  at    (Coroutine boundary. ( )
  at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)

I'm using Ktor client with an MPP project.

After upgrading to 1.5.2, I started having this error on a yarn project that is using my js module as a javascript dependency:

./node_modules/mp-game-clib/mp-game-clib.js
Module not found: Can't resolve 'node-fetch' in '/Users/brunomed/git/pok-p2p-game-demo/react-app/node_modules/mp-game-clib'

That's the relevant Gradle config for the library project:

    js(IR) {
        useCommonJs()
        browser {
            binaries.library()
            webpackTask {
                enabled = false
            }
            testTask {
                useKarma {
                    useChromeHeadless()
                    webpackConfig.cssSupport.enabled = true
                }
            }
        }
    }
    ...
    }
    sourceSets {
        val commonMain by getting {
            dependencies {
                implementation(project(":client-server"))
                implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.1.0")

                implementation("io.ktor:ktor-client-core:$ktorVersion")
                implementation("io.ktor:ktor-client-websockets:$ktorVersion")
                implementation("com.benasher44:uuid:0.2.3")

                // https://youtrack.jetbrains.com/issue/KTOR-541
                runtimeOnly(npm("text-encoding", "0.7.0"))
                runtimeOnly(npm("abort-controller", "3.0.0"))
                runtimeOnly(npm("node-fetch", "2.6.1"))                           <== Had to add this to fix the problem for now, I didn't need it with Ktor 1.5.1 
            }
        }
    ...
    }

Not sure if I'm doing something wrong here, but adding those npm dependencies is the only way I could find to produce correct package.json on the jsBrowserProductionLibraryDistribution goal (previous comment here: https://youtrack.jetbrains.com/issue/KTOR-541#focus=Comments-27-4731438.0-0)

Documentation for FormPart says @param value content, could be [String], [Number] or [Input] https://github.com/ktorio/ktor/blob/e03bafda3b3d72fcac166e46cf55e5d2d9383660/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/formDsl.kt#L17 But later in this file Input is deprecated

   @Suppress("UNUSED_PARAMETER")
    @Deprecated(
        "Input is not reusable. Please use [InputProvider] instead.",
        level = DeprecationLevel.ERROR,
        replaceWith = ReplaceWith("appendInput(key, headers) { /* create fresh input here */ }")
    )
    public fun append(key: String, value: Input, headers: Headers = Headers.Empty) {
        error("Input is not reusable. Please use [InputProvider] instead.")
    }

And direct usage will cause exception

public fun formData(vararg values: FormPart<*>): List<PartData> {
...
    is Input -> error("Can't use [Input] as part of form: $value. Consider using [InputProvider] instead.")
...
}

The function channel.readRemaining sometimes returns a packet (possibly the empty one) when the channel is closed abnormally.

Changelog https://github.com/Kotlin/kotlinx.coroutines/releases/tag/1.4.3

A user gets the unresolved reference "call.receiveText" error when trying to use examples from https://ktor.io/docs/requests.html

There are reports that the tutorial located at https://play.kotlinlang.org/hands-on/Creating%20HTTP%20APIs%20with%20Ktor/02_application-init does not work well with the latest version of the Ktor plugin. We recently ported this tutorial over to our docs. Please make sure it is all correct.

Right now the page https://ktor.io/clients/http-client/features/auth.html is redirected to https://ktor.io/docs/http-client-features.html but should be redirected to https://ktor.io/docs/features-auth.html instead.

The first link from the comment https://github.com/ktorio/ktor/issues/507#issuecomment-410173950 gives a 404 page.

Most contain bintray and jcenter repositories. They need to be removed.

Some snippets have 4 spaces and some 2

https://ktor.io/docs/serving-static-content.html#serving-individual-files

Should be: Gradle Kotlin, Gradle Groovy, Maven

The wizard generates a new app that compiles and runs, but when clicking on the default route as indicated by the output, it doesn't respond. I think we should provide a default route not only as means of folks knowing that it works, but also as a starting point.

Wizard is still referencing Bintray and jcenter which aren't required.

repositories {
    mavenLocal()
    jcenter()
    maven { url = uri("https://kotlin.bintray.com/ktor") }
}

To reproduce the issue, create a new Ktor project with the Maven build system. The pom.xml file will contain the following code for the exec-maven-plugin plugin:

                <configuration>
                    <mainClass>io.ktor.server.netty.DevelopmentEngine</mainClass>
                </configuration>

But DevelopmentEngine is deprecated.

The 1.5.x update broke the auto-reload feature. I made an example repo here, when switching from 1.4.x to 1.5.x the auto-reload break (not working anymore). You can reproduce by running ./gradlew run and ./gradlew build -t and switch versions.

Could not find artifact org.jetbrains.kotlin:kotlin-maven-serialization:pom:${kotlin.version} in central (https://repo.maven.apache.org/maven2)

I assume this is the offending block:

<dependency>
    <groupId>org.jetbrains.kotlin</groupId>
    <artifactId>kotlin-maven-serialization</artifactId>
    <version>${kotlin.version}</version>
</dependency>

If we're linking to the specific code for a feature on GitHub (which should probably be indicated in the metadata for the feature and only linked if present), it should link to the specific feature. Currently all the ones we ship out of the box point to generic https://github.com/ktorio/ktor

Also, please note the text should say

"Source on GitHub" instead of "See features's GitHub"

The feature generates the following code in resources/static/index.html:

<html>
<head>
</head>
<body>
<h1>Hello Ktor!<h1>
</body>
</html>

As IntelliJ IDEA notes correctly, the h1 tag is not being closed properly. Should be </h1>.

With a barebones Ktor project as generated by the Wizard, the stubbed-out code snippets in the README.md cause IntelliJ to believe we're committing very faulty software when importing our project into Git (and, likely, other VCS). IJ shows 50+ errors (not warnings).

This looks very scary, and makes it feel like something is broken with the setup even when there isn't. Please either annotate the README with some kind of flag to let IntelliJ know not to check all the code fences, or otherwise exclude the README file from code analysis.

Tested on:

IntelliJ IDEA 2021.1 Beta (Ultimate Edition)
Build #IU-211.6305.21, built on March 4, 2021
IntelliJ IDEA EAP User
Expiration date: April 4, 2021
Runtime version: 11.0.10+9-b1341.18 x86_64
VM: Dynamic Code Evolution 64-Bit Server VM by JetBrains s.r.o.
macOS 10.14.6
GC: G1 Young Generation, G1 Old Generation
Memory: 4096M
Cores: 8
Registry: ide.images.show.chessboard=true
Non-Bundled Plugins: com.jetbrains.darkPurpleTheme (1.2), org.nik.presentation-assistant (1.0.9), wu.seal.tool.jsontokotlin (3.6.1), org.asciidoctor.intellij.asciidoc (0.32.26), stardust (1.5.2105), com.jetbrains.intellij.api.watcher (6.46.0), intellij.ktor (1.5.1-eap-1)
Kotlin: 211-1.4.21-release-IJ6305.1

plugins {
    id 'application'
    id 'org.jetbrains.kotlin.jvm' version '1.4.10'
    id "org.jetbrains.kotlin.plugin.serialization" version "1.4.10" // <== these should probably also be single quotes – no need to switch.
}

import org.jetbrains.kotlin.gradle.dsl.Coroutines
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

If I create a project with ContentNegotiation, Gson, and Routing, I don't get a run configuration automatically configured. I need to press the green gutter play icon, or run the run Gradle task.

Adding a bunch more features seems to sometimes include a default "ApplicationKt" run configuration.

For some reason it also add numerous entries

<repositories>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

1. Create a new Ktor project.
2. Go to the Features page and find the Micrometer Metrics feature.
3. Find the [monitoring the JVM][micrometer_jvm_metrics] text => should be a link

1. Create a new Ktor project.
2. Go to the Features page and find the Micrometer Metrics feature.
3. Click any link => doesn't work

When I scroll feature documentation to some point and then click to another feature the scroll position remains the same. This requires me to scroll to the top to start reading another feature's documentation.

As a Ktor backend developer I want to have an IDE interface with endpoints list, generated from my routing configuration: Possible features of endpoints map:

• go to the related code;
• generate a new route wizard;
• open in browser;
• generate dummy request with applicable params (according to data class) for IDEA http-client;
• generate test from request/response;
• show pipelines and interceptors chain;
• show execution timings

Main goals

1. Make API easy to understand for users: a) Get rid of concept of pipelines and phases b) Make API compact (reduce number of options available, i.e. consider smaller number of "phases") c) Rework naming to make it intuitive d) Think about defaults and yet support common case
2. API must be compatible with old existing features
3. All usages of old features rewritten with new API must remain unchanged yet working. This means all import {...} blocks should still work for features rewritten to the new API as well as with the old features.
4. Choosing "phase" must become intuitive and easy
5. Old API functionality should remain available for experienced users.
6. API should fit into Kotlin best patterns and concepts (DSL is a good option here)

Onboarding (first steps)

Old API

Let's take a look at some small feature that was defined with the old Features API. This example was taken from our official documentation:

class CustomFeature(configuration: Configuration) {
    val prop = configuration.prop // Copies a snapshot of the mutable config into an immutable property.

    class Configuration {
       var prop = "value" // Mutable property.
    }

    // Implements ApplicationFeature as a companion object.
    companion object Feature : ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature> {
       // Creates a unique key for the feature.
       override val key = AttributeKey<CustomFeature>("CustomFeature")

       // Code to execute when installing the feature.
       override fun install(pipeline: ApplicationCallPipeline, configure: Configuration.() -> Unit): CustomFeature {

           // It is responsibility of the install code to call the `configure` method with the mutable configuration.
           val configuration = CustomFeature.Configuration().apply(configure)

           // Create the feature, providing the mutable configuration so the feature reads it keeping an immutable copy of the properties.
           val feature = CustomFeature(configuration)

           // Intercept a pipeline.
           pipeline.intercept(…) {
                // Perform things in that interception point.
           }
           return feature
       }
    }
}

Old Problems:

• This code does not look welcoming to newcomers at all.
• It is not easy to understand.
• It contains a lot of concepts to deal with that are not hidden by the API: ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature>, AttributeKey<CustomFeature>, ApplicationCallPipeline, pipeline.intercept
• It is not clear where is the feature itself and where is it's configuration: there is a CustomFeature that has some properties, and there is also a Configuration class that defines configuration, and there is also a Feature object. It is not 100% obvious how each of them is used and such model confuses unexperienced readers because there is no clear separation between these concepts.

Proposed solution:

1. Instead of creating 2 nested classes and an object, extending some weird class ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature> now it is sufficient to just create an instance of KtorFeature using createFeature { ... } DSL method, and to define a configuration class for it separately:

class Configuration {
    // Properties and configuration functions go here:
    ...
}

val CustomFeature = createFeature(name = "CustomFeature", createConfiguration = ::Configuration) {
    // What feature does goes here:
    ...
}

[update]: cc @leonid.stashevsky createFeature block can be basically called anywhere as long as it makes sense for it to be visible by the scope of install that uses it's result. In other words, createFeature block creates an object (and it makes sense to just make it a global object) of type KtorFeature and this object can be used as an argument of install(...).

This feature can now be used as it was before, with just calling install and configuring it in your main application code:

install(CustomFeature) {
    // set mutable config properties
}

Benefits

• Now, users will see much more clear separation between the data and actions, and they will not have to deal with confusing methods and base classes to extend.
• It is fully compatible with the old code that uses features

Extending Ktor HTTP behaviour

Old API

In the old API users had to deal with a concept of phases and pipelines. Below there is a list of them:

• ApplicationCallPipeline
  • Setup : Phase for preparing call and it's attributes for processing
  • Monitoring : Phase for preparing call and it's attributes for processing
  • Features : Phase for features. Most features should intercept this phase.
  • Call : Phase for processing a call and sending a response
  • Fallback : Phase for handling unprocessed calls
• ApplicationSendPipeline
  • Before : The earliest phase that happens before any other
  • Transform : Transformation phase that can proceed with any supported data like String
  • Render : Phase to render any current pipeline subject into [io.ktor.http.content.OutgoingContent], beyond this phase only [io.ktor.http.content.OutgoingContent] should be produced by any interceptor
  • ContentEncoding : Phase for processing Content-Encoding, like compression and partial content
  • TransferEncoding : Phase for handling Transfer-Encoding, like if chunked encoding is being done manually and not by engine
  • After : The latest application phase that happens right before engine will send the response
  • Engine : Phase for Engine to send the response out to client.
• ApplicationReceivePipeline
  • Before : Executes before any transformations are made
  • Transform : Executes transformations
  • After : Executes after all transformations

Note, that in the list above phases are ordered!

Also, each feature had an ability to create it's custom phase in any of the 3 pipelines, and insert this phase anywhere in the order of existing phases. In order to use some phase ("intercept a pipeline in a specific phase") a feature was meant to call pipeline.intercept method on a specific phase.

Here is an example of usage of the old API taken from the MicrometerMetrics implementation:

 override fun install(pipeline: Application, configure: Configuration.() -> Unit): MicrometerMetrics {
    ...
    val phase = PipelinePhase("MicrometerMetrics")
    pipeline.insertPhaseBefore(ApplicationCallPipeline.Monitoring, phase)

    pipeline.intercept(phase) {
        // some actions go here
    }
    val postSendPhase = PipelinePhase("MicrometerMetricsPostSend")
    pipeline.sendPipeline.insertPhaseAfter(ApplicationSendPipeline.After, postSendPhase)
    pipeline.sendPipeline.intercept(ApplicationSendPipeline.After) {
        // some actions go here
    }

Old problems:

• There too many phases and pipelines (3 pipelines with 15 phases total)
• Not obvious which phase to select,
• No defaults for some simple usage (onboarding)
• With user-defined phases and insertBefore/insertAfter it builds a complex mental model of the global order of phases and requires a deep knowledge about phases each feature uses in order to think about the order of execution.
• Concept of phases and pipelines is not intuitive and does not mentally map to the concept of Routing in Ktor. It is not obvious when each phase is executed and what are the pipelines.

Proposed solution:

First of all, there should be simple defaults that would extend basic functionalities that users see in Ktor. There will be 3 "default" methods that are mapped by name with what call inside a routing block each of them extend:

• onCall : extends what happens when a server is processing any HTTP method call
• onReceive : extends behaviour of the server when user writes call.receive() in a route { ... } block
• onRespond : extends behaviour of the server when user writes call.respond() (call.respondText(), etc.) in a route { ... } block

In code they should be placed inside createFeature block:

val F = createFeature(...) {
    onCall {
        ...
    }
    onReceive {
        ...
    }
    onRespond {
        ...
    }
}

Such defaults are useful for newbies because they are meant to provide the most commonly used functionality without a requirement to know a lot about other options.

Also, for some more advanced use cases there will be more advanced builders with more options to extend:

1. extendCallHandling { ... } -- a block that configures what happens when HTTP method call handling

   • monitoring { ... } -- defines actions to perform before the call was processed by any feature (including Routing). It is useful for monitoring and logging (see CallLogging feature) to be executed before any "real stuff" was performed with the call because other features can change it's content as well as add more resource usage etc. while for logging and monitoring it is important to observe the pure (untouched) data.
   • onCall { ... } -- same as onCall outside of extendCallHandling. Most used extension. It defines what to do with the call that is being currently processed by a server.
   • fallback -- defines what to do with the call in case request was not handled for some reason. Usually, it is handy to use fallback { call -> ...} to set the response with some message or status code, or to throw an exception.

2. extendReceiveHandling supports following callbacks:

   • beforeReceive — defines actions to perform before any transformations were made to the received content. It is useful for caching.
   • onReceive — defines how to transform data received from a client.

3. extendResponseHandling supports following callbacks:

   • beforeResponse — allows to use the direct result of call processing (see onCall)
   • onResponse — does transformations of the data. Example: you can write a custom serializer wit h it.
   • afterResponse — allows to calculate some stats on the data that was sent, or to handle errors. It is used in Metrics, CachingHeaders, StatusPages.

Example:

val F = createFeature(...) {
    onCall {
        // prepare some data
    }
    extendResponseHandling {
        beforeResponse {
            // use this data
        }
        onResponse {
            // transform data to a string (i.e. serialize data)
        }
    }
}

Benefits:

• Now we provide fast-to-access defaults that newbies can use (now it is obvious what to select from: not sure -- use a default)
• There are much less options to select from, and these options are mapped to what users do when they define handling of routes
• Api doesn't introduce any new complex or uncommon concepts (like phases or pipelines, or interceptors)

Managing order of execution between features

Old API

In the old API there was a concept of phases that one could introduce and then insert anywhere with insertPhaseBefore / insertPhaseAfter, and then intercept these phases. One of the most common things it was used for was creating order dependencies between features. This idea is explained here: https://www.ximedes.com/2020-09-17/role-based-authorization-in-ktor/. This was useful to either prepare some data that can be used later by another feature, or to use some data already produced by some other feature.

Old Problems:

• It required using 3 concepts to do 1 thing: use "phase", "insertPhase" and "intercept" in order to "execute one feature before/after another"
• It required explicitly knowing which phases another feature was using.

Proposed solution:

Each feature now internally remembers which phases in which pipelines it intercepts (of course, without revealing this complex and confusing information to users). And createFeature block also provides following functions inside:

• beforeFeature(other) { ... }
• afterFeature(other) { ... }

Example:

val F = createFeature(...) {
    ...
}

val G = createFeature(...) {
    beforeFeature(F) {
        onCall {
            // prepare some data for feature F
        }
    }
}

Benefits:

Now the API just does what it says: it is simply executing one feature before/after another without introducing any complex entities and without a requirement to manually manage the internals.

Manually managing features execution:

Old API

In the old API it was possible to manually manage execution of the current pipeline. Inside any interceptor it was possible to call the following functions of PipelineContext<, >:

public interface PipelineContext<TSubject : Any, TContext : Any> : CoroutineScope {
    /**
     * Object representing context in which pipeline executes
     */
    public val context: TContext

    /**
     * Subject of this pipeline execution that goes along the pipeline
     */
    public val subject: TSubject

    /**
     * Finishes current pipeline execution
     */
    public fun finish()

    /**
     * Continues execution of the pipeline with the given subject
     */
    public suspend fun proceedWith(subject: TSubject): TSubject

    /**
     * Continues execution of the pipeline with the same subject
     */
    public suspend fun proceed(): TSubject
}

Also, there were some extension methods over this class:

• application : Current application for the context
• call : Current call for the context
• transformDefaultContent : Default outgoing content transformation

Although it introduced really complex concept of subject to understand, it can be required for some advanced users and we can't get rid of it completely. One of the most popular patterns that use PipelineContext is the following (taken from VelocityFeature):

pipeline.sendPipeline.intercept(ApplicationSendPipeline.Transform) { value ->
    if (value is VelocityContent) {
        val response = feature.process(value)
        proceedWith(response)
    }
}

It actually allows to write a custom data serializer for a specific type (VelocityContent in this example). proceedWith(outgoingContent) allows to continue the execution of our HTTP pipeline with the data serialized to a resulting outgoingContent.

Old problems

• Really complex and hard to understand but yet useful
• Some members/extensions are not that popular or not required for the API (application is not commonly used, context makes no sense to newbies)
• proceedWith and suject are useful but are mostly used for serialization features
• serialization use case requires knowing about OutgoingContent type that is not mapped to any general knowledge, and in fact there is always a default serializer from String, ByteChanel and ByteArray types to the OutgoingContent.

Proposed solution:

1. For the case of serialization we will introduce following simple API that users may default to:

public fun <T : Any> KtorFeature<T>.serializeDataToString(callback: suspend SendExecution.(Any) -> String?): Unit =
    onSend { callback(subject)?.let { proceedWith(it) } }

public fun <T : Any> KtorFeature<T>.serializeDataToBytes(callback: suspend SendExecution.(Any) -> ByteArray?): Unit =
    onSend { callback(subject)?.let { proceedWith(it) } }

public fun <T : Any> KtorFeature<T>.serializeDataToChanel(callback: suspend SendExecution.(Any) -> ByteWriteChannel?): Unit =
    onSend { callback(subject)?.let { proceedWith(it) } }

This API can be used as following:

private class Formula(val x: String, val execute: (String) -> String)

private val F = createFeature("F", {}) {
    serializeDataToString { f ->
        // return null for irrelevant data types
        if (f !is Formula) return@serializeDataToString null

        // Do a transformation for the current data type
        f.execute(f.x)
    }
}

private fun Application.f() {
    routing {
        get("/") {
            call.respond(Formula("cake") { x -> "x * x + 1".replace("x", x) })
        }
    }
}

Also, for any more complex use cases we will introduce the following wrapper over PipelineContext<SubjectT, ApplicationCall>. It is meant to hide unneeded methods and extensions without limiting functionality too much:

public inline class Execution<SubjectT : Any>(private val context: PipelineContext<SubjectT, ApplicationCall>) {
    public suspend fun proceed(): SubjectT = context.proceed()
    public suspend fun proceedWith(subectT: SubjectT): SubjectT = context.proceedWith(subectT)
    public fun finish(): Unit = context.finish()
    public val subject: SubjectT get() = context.subject
    public val call: ApplicationCall get() = context.call
...

[update] cc @leonid.stashevsky Also following methods can be added to handle application-level properties from a feature:

    // Useful methods
    public val environment: ApplicationEnvironment get() = context.application.environment
    public val configuration: ApplicationConfig get() = environment.config
    public val port: Int get() = configuration.propertyOrNull("ktor.deployment.port")?.getString()?.toInt() ?: 8080
    public val host: String get() = configuration.propertyOrNull("ktor.deployment.host")?.getString() ?: "0.0.0.0"
    // Following calback can be used to close resources that a feature has allocated (if needed):
     public fun onShutdown(callback: suspend () -> Unit) {
        GlobalScope.launch(context.coroutineContext) {
            callback()
        }
    }
}

Also there are type aliases for executions of the different stages:

public typealias CallExecution = Execution<Unit>
public typealias ReceiveExecution = Execution<ApplicationReceiveRequest>
public typealias SendExecution = Execution<Any>

Each of the methods has a corresponding Execution type as a reciever for a callback, for example:

public override fun onCall(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit { ... }

This means, that inside a feature definition advanced users who know what they do may use proceed, finish etc. methods anywhere in the code:

val F = createFeature(...) {
    extendRespondHandling {
        beforeRespond {
            ...
            if (someCondition)
                finish()
            ...
        }
    }
}

[update] cc @leonid.stashevsky Another option for an API

There are following minor issues with the proposed API:

• Naming is quite long for advanced options (extendCallHandling as opposed to onCall), although it is made so intentionally to hide advanced settings from newbies by a longer name.
• It sometimes requires writing thisin Intellij in order to call a completion to learn what's next.

Benefits

• Now it is easy to write serializers
• Now advanced users can still use what they want
• No complex classes are in your face when you write simple code, so nothing will confuse newbies.

Alternative

We can introduce following interfaces (usage will be described later):

public interface OnCall {
    public operator fun invoke(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
    public fun monitoring(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
    public fun fallback(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
}

public interface OnReceive {
    public operator fun invoke(callback: suspend ReceiveExecution.(ApplicationCall) -> Unit): Unit
    public fun before(callback: suspend ReceiveExecution.(ApplicationCall) -> Unit): Unit
}

public interface OnResponse {
    public operator fun invoke(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
    public fun before(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
    public fun after(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
}

And change FeatureContext (a supertype of KtorFeature) to the following:

public interface FeatureContext {
    public val onCall: OnCall
    public val onReceive: OnReceive
    public val onResponse: OnResponse
}

After implementing FeatureContext it will allow us to write the code in the next way:

val F = makeFeature("F", {}) {

    onCall { ... }

    onCall.fallback { ... }
    onCall.monitoring { ...}

    beforeFeature(G) {
        onCall { ... }
        onReceive.before { ... }
        onResponse.after { ... }
    }
    ...

So, in general, following API transformations have been applied:

• onCall { ... } -> onCall { ... }
• extendCallExecution { fallback { ... } } -> onCall.fallback { ... }
• extendCallExecution { monitoring { ... } } -> onCall.monitoring { ... }
• onRespond { ... } -> onRespond { ... }
• extendRespondExecution { before { ... } } -> onRespond.before { ... }
• extendRespondExecution { after { ... } } -> onRespond.after { ... }
• onReceive { ... } -> onReceive { ... }
• extendReceiveExecution { before { ... } } -> onReceive.before { ... }

Benefits

• Much shorter and readable names
• Advanced functionality is still hidden (now it is hidden by the dot .)
• Completion perfectly works when one types a dot after an object name to see the options
• Simple things are kept the same because onCall, onReceive and onSend are executable objects now

Disadvantages

• Not conventional way of writing API, may be uncommon to write dots after kinda-function names

[update] cc @leonid.stashevsky Compatibility with features written in old API

New API provides an ability for features to interact/depend on each other with beforeFeature and afterFeature.

Inside it automatically creates everything that is needed to guarantee the order between features. But this change requires old features to imlement new interface InterceptionsHolder in order to make it working with the old features. We can easily do that for builtin features:

/**
 * Compatibility class. Interception class for Call phase
 * */
public typealias CallInterception = Interception<Unit>
/**
 * Compatibility class. Interception class for Receive phase
 * */
public typealias ReceiveInterception = Interception<ApplicationReceiveRequest>
/**
 * Compatibility class. Interception class for Send phase
 * */
public typealias ResponseInterception = Interception<Any>

/**
 * Compatibility class. Every feature that needs to be compatible with `beforeFeature(...)` and `afterFeature(...)`
 * needs to implement this class. It defines a list of interceptions (see [Interception]) for phases in different pipelines
 * that are being intercepted by the current feature.
 * 
 * It is needed in order to get first/last phase in each pipeline for the current feature and create a new phase
 * that is strictly earlier/later in this pipeline than any interception of the current feature.
 * 
 * Note: any [KtorFeature] instance automatically fills these fields and there is no need to implement them by hand.
 * But if you want to use old feature API based on pipelines and phases and to make it available for `beforeFeature`/
 * `afterFeature` methods of the new features API, please consider filling the phases your feature defines by implementing
 * this interface [InterceptionsHolder].
 * */
public interface InterceptionsHolder {
    public val name: String get() = this.javaClass.simpleName

    public val fallbackInterceptions: MutableList<CallInterception>
    public val callInterceptions: MutableList<CallInterception>
    public val monitoringInterceptions: MutableList<CallInterception>

    public val beforeReceiveInterceptions: MutableList<ReceiveInterception>
    public val onReceiveInterceptions: MutableList<ReceiveInterception>

    public val beforeResponseInterceptions: MutableList<ResponseInterception>
    public val onResponseInterceptions: MutableList<ResponseInterception>
    public val afterResponseInterceptions: MutableList<ResponseInterception>

    public fun newPhase(): PipelinePhase = PipelinePhase("${name}Phase${Random.nextInt()}")
}

Question: should we make it internal and hide

pros:

• It reveals old API abstactions and is really complex
• It has a lot of fields to implement and is confusing
• We can implement it in our builtin features and this will cover almost all the cases of the features interaction. It is very rare when someone wants to insert their feature before/after some other third-party feature while it is much more common to insert one relatively to some builtin feature. For the rest of the cases, we can suggest using new API that is easier and does all the work automatically.

cons:

• (Didn't actually find any use-case) probably we'll force old users to rewrite their features with the new API to make them available to the new features.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1656

Ktor Version and Engine Used (client or server and name) 1.3.0

Describe the bug CORS can't pass on some none standard orgin like chrome-extension://mfgdmpfihlmdekaclngibpjhdebndhdj

To Reproduce open some new page in chrome or other browser,just call any api,CORS throw would error(brower side).

Expected behavior should be AnyOrigin pass by CORS, when we config it as host("*") or anyHost()

Hi team, I'm using as Jackson content negotiation feature and can't override Kotlin module configuration when installing using jackson function. Here's my code:

install(ContentNegotiation) {
  jackson {
    registerModule(kotlinModule { nullIsSameAsDefault(true) }) // doesn't work ❌
    registerModule(KotlinModule(nullIsSameAsDefault = true)) // also doesn't work ❌
  }
}

When I trace to ktor-jackson source code, I found this code:

fun jacksonObjectMapper(): ObjectMapper = jsonMapper { addModule(kotlinModule()) }

Look like Kotlin module already added and there is no way to reconfigure it (?). I can't get any information in issues and documentation page.

My temporary workaround,

install(ContentNegotiation) {
  val mapper = jsonMapper { addModule(KotlinModule(nullIsSameAsDefault = true)) } // work great ✔
  val jackson = JacksonConverter(mapper)
  register(ContentType.Application.Json, jackson)
}

Thank you!

kotlin: 1.4.20 kotlinx.coroutines: 1.4.2

Sessions feature is configured in a rather standard manner:

        install(Sessions) {
            cookie<UserIdPrincipal>(
                Cookies.SESSION_ID,
                storage = directorySessionStorage(sessionStorageRootDir)
            ) {
                cookie.path = "/"
                cookie.httpOnly = true
                cookie.extensions["SameSite"] = "Lax"
            }
        }

When it is called during a pipeline execution, an unexpected exception is thrown.

java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
	at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	(Coroutine boundary) ~[?:?]
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:259) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) ~[ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) ~[ktor-server-netty-jvm-1.4.3.jar:1.4.3]
Caused by: java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
	at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.4.20.jar:1.4.20-release-308 (1.4.20)]
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:342) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:27) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:109) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt.async(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt.async$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:37) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:31) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_251]
	at io.ktor.sessions.BaseCache.getOrCompute(Cache.kt:36) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache.getOrCompute(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseTimeoutCache.getOrCompute(Cache.kt:163) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:63) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:43) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:34) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

Code snippet:

    val server = embeddedServer(Netty, env) {
        configureBootstrap = {
            group(NioEventLoopGroup(16, Executors.newCachedThreadPool()))
        }
    }

During NettyApplicationEngine initialization we have:

    private val bootstraps = environment.connectors.map { connector ->
        ServerBootstrap().apply {
            configuration.configureBootstrap(this)
            group(connectionEventGroup, workerEventGroup)

The call to group(...) tries to initialize the parent and child EventLoopGroups for a second time, which throws the exception above.

Periodically (not every time) during running tests in https://github.com/rsocket/rsocket-kotlin using ktor-io exception appeared trying to build ByteReadPacket using buildPacket {}

Exception in thread "DefaultDispatcher-worker-9 @coroutine#122840" io.ktor.utils.io.core.InsufficientSpaceException: Not enough free space to write long integer of 8 bytes, available 7 bytes.
	at io.ktor.utils.io.core.BufferPrimitivesKt.writeLong(BufferPrimitives.kt:1081)
	at io.ktor.utils.io.core.OutputPrimitivesKt.writeLongFallback(OutputPrimitives.kt:50)
	at io.ktor.utils.io.core.OutputPrimitivesKt.writeLong(OutputPrimitives.kt:45)
	at io.rsocket.kotlin.frame.KeepAliveFrame.writeSelf(KeepAliveFrame.kt:33)
	at io.rsocket.kotlin.frame.Frame.toPacket(Frame.kt:36)
	at io.rsocket.kotlin.internal.RSocketState$start$3.invokeSuspend(RSocketState.kt:143)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.selects.SelectBuilderImpl.resumeWith(Select.kt:306)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

CommandLine.kt

requireNotNull(getKey(sslKeyAlias, sslPrivateKeyPassword.toCharArray()) == null)

is always successful

Multiple instance of "has been" in exception messages such as

Request timeout has been expired 

https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreLinuxJava11/22298?

https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreLinuxJava8/21950

https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixStressTestLinuxJava8/22168?

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/821

Hi Ktor team!

I have some problems with the client when make many parallels http requests.

I hope logs and code examples will help you.

Ktor Version

1.0.1

Ktor Engine Used

Android Engine

Features enabled

JsonFeature with GsonSerializer

Platform and hardware

Android 7, 4-core ARM CPU

Feedback

fun test() {
    for (i in 1..100) {
        println("Start launch")
        GlobalScope.launch(Dispatchers.Default) {
            val data = client.get<MyResponse>(url)
            println("Response completed: $i")
        }
        println("Finish launch")
    }
}`

Logs (please see time, it important, in really one request take about 1-3 seconds): ` 2018-12-24 01:07:47.388 I/System.out: Start launch 2018-12-24 01:07:47.403 I/System.out: Finish launch

// skiped logs: started 100 coroutines

2018-12-24 01:07:47.487 I/System.out: Start launch 2018-12-24 01:07:47.488 I/System.out: Finish launch // last coroutine started

// there is a big pause: in really one request take about 1-3 seconds

2018-12-24 01:07:57.671 I/System.out: Response completed: 9 // first response

// there is a big pause: in really request take about 1-3 seconds

2018-12-24 01:08:27.280 I/System.out: Response completed: 94 // second response 2018-12-24 01:08:28.265 I/System.out: Response completed: 1 2018-12-24 01:08:28.275 I/System.out: Response completed: 2 2018-12-24 01:08:28.285 I/System.out: Response completed: 4 2018-12-24 01:08:28.310 I/System.out: Response completed: 8 2018-12-24 01:08:28.396 I/System.out: Response completed: 80

// skiped logs: responses are distributed evenly in time

2018-12-24 01:08:29.268 I/System.out: Response completed: 92 2018-12-24 01:08:29.280 I/System.out: Response completed: 97 2018-12-24 01:08:29.296 I/System.out: Response completed: 99 2018-12-24 01:08:29.371 I/System.out: Response completed: 100 // last response`

I overrided GsonSerializer and added logs into problematic code.

override suspend fun read(type: TypeInfo, response: HttpResponse): Any {
    println("Start response.readText on ${Thread.currentThread().name}")
    val text = response.readText()
    println("End response.readText on ${Thread.currentThread().name}")
    return backend.fromJson(text, type.reifiedType) ?: Any()
}

Logs:

`2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-3 // first start readText 2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-1 2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:32:40.160 I/System.out: Start response.readText on DefaultDispatcher-worker-1

// skiped 45 logs Start response.readText on ..

2018-12-24 01:33:11.137 I/System.out: End response.readText on DefaultDispatcher-worker-3 // first end readText 2018-12-24 01:33:11.311 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:12.202 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:12.298 I/System.out: Start response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:12.663 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:12.668 I/System.out: Start response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:14.052 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:14.091 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:14.238 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:14.313 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.475 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.486 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.576 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.646 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:15.746 I/System.out: End response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.754 I/System.out: End response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.766 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:15.877 I/System.out: End response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:15.914 I/System.out: End response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.930 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:15.938 I/System.out: End response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.995 I/System.out: End response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:16.051 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:16.101 I/System.out: End response.readText on DefaultDispatcher-worker-2

// skiped many End response.readText on ..

2018-12-24 01:33:16.963 I/System.out: End response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:16.963 I/System.out: End response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:16.974 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:16.982 I/System.out: End response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:17.115 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:17.129 I/System.out: End response.readText on DefaultDispatcher-worker-3 // last end readText`

I think problem in content.readRemaining method.

Maybe related https://github.com/ktorio/ktor/issues/768, https://github.com/ktorio/ktor/issues/770

I am ready to provide more details let me know.

Thanks!

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1266

Ktor Version and Engine Used (client or server and name) Ktor 1.2.3 embedded Jetty server.

Describe the bug If server is started with "user-local default: JAR manifest" command line shortener it seems allUrls in ApplicationEngineEnvironmentReloading.kt only has a couple of java jars thus it prints:

2019-08-07 14:11:13,088 INFO  --- [main] Application : No ktor.deployment.watch patterns match classpath entries, automatic reload is not active

To Reproduce Error is shown on startup with the mentioned shorten command line option.

Expected behavior Reloading works. It works for us if we use the classpath file shorten command line option. The options are not visible for kotlin run configurations but can be accessed in a Java Application run application (with same options as a Kotlin version.). I think the broken option is the default so it would be much nicer if Ktor worked with that.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1657

Subsystem Server

Is your feature request related to a problem? Please describe. We wanted our typed locations to be able to define a permission node / access level. We did this via an additional annotation but then we had to write an interceptor that would get the class annotated with @Location from the request path and check for ours. The problem is that there doesn't seem to be any public access to the mapping done in Locations. Right now we're using a pretty-narly looking reflection solution that's less than ideal.

Describe the solution you'd like Exposing io.ktor.locations.Locations$LocationInfo and an immutable copy of io.ktor.locations.Locations.info should be enough.

Motivation to include to ktor I'm sure this could be a beneficial feature to a lot of users, even if it's not implemented in the way described above. It'll allow users to add additional properties to their routes and implement them in interceptors (and elsewhere) without having to do any reflection or map what is already mapped within ktor.locations.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1747

Hi all I've created the Ktor project in Intellj with Ktor plugin, when running the application it shows the following error message:

020-03-23 20:31:20.192 [main] INFO  Application - No ktor.deployment.watch patterns specified, automatic reload is not active
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method.callMethod(CallerImpl.kt:97)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method$Static.call(CallerImpl.kt:106)
	at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:166)
	at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:110)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.callFunctionWithInjection(ApplicationEngineEnvironmentReloading.kt:384)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:330)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1$$special$$inlined$forEach$lambda$1.invoke(ApplicationEngineEnvironmentReloading.kt:275)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1$$special$$inlined$forEach$lambda$1.invoke(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:310)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:274)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartup(ApplicationEngineEnvironmentReloading.kt:290)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:272)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:125)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:245)

The java version, that is installed on my computer:

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (Zulu 8.44.0.11-CA-linux64) (build 1.8.0_242-b20)
OpenJDK 64-Bit Server VM (Zulu 8.44.0.11-CA-linux64) (build 25.242-b20, mixed mode)

And here in the Application.kt:

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)

@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(Authentication) {
    }

    install(ContentNegotiation) {
        gson {
        }
    }

    val client = HttpClient() {
        install(JsonFeature) {
            serializer = GsonSerializer()
        }
        install(Logging) {
            level = LogLevel.HEADERS
        }
        BrowserUserAgent() // install default browser-like user-agent
        // install(UserAgent) { agent = "some user agent" }
    }
    runBlocking {
        // Sample for making a HTTP Client request
        /*
        val message = client.post<JsonSampleClass> {
            url("http://127.0.0.1:8080/path/to/endpoint")
            contentType(ContentType.Application.Json)
            body = JsonSampleClass(hello = "world")
        }
        */
    }

    routing {
        get("/") {
            call.respondText("HELLO WORLD!", contentType = ContentType.Text.Plain)
        }

        get("/json/gson") {
            call.respond(mapOf("hello" to "world"))
        }
    }
}

data class JsonSampleClass(val hello: String)

Thanks

Ktor 1.5.3 relies on kotlinx-html-jvm:0.7.3, but version 0.7.3 has not been published to any of the public maven repos.

https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreMacOSJava8/22194?

Ktor 1.5.0 I am using CIO Engine which has a default timeout of 15000 milliseconds, however, I changed the timeout for a particular request using the HttpRequestBuilder.timeout extension function, but I still get ConnectTimeoutException after 15000 milliseconds.

Gradle:

dependencies {
    ....
   def ktorVersion = "1.5.0"
    implementation "io.ktor:ktor-client-core:$ktorVersion"
    implementation "io.ktor:ktor-client-cio:$ktorVersion"
    implementation "io.ktor:ktor-client-serialization:$ktorVersion"
    implementation "io.ktor:ktor-client-logging-jvm:$ktorVersion"
    ....
}

HttpClient:

HttpClient(CIO) {
    defaultRequest {
        host = HOST
        header("Content-Type", "application/json")
    }

    Logging {
        logger = object : Logger {
            override fun log(message: String) {
                Timber.i(message)
            }
        }
        level = LogLevel.ALL
    }
    install(HttpTimeout)
}

Request:

httpClient.post("${HOST_URL}/users/new") {
     timeout {
         requestTimeoutMillis = 30000
     }
     body = requestBody
}

STR: perform http GET with ktor Apache client to a resource which replies with truncated content

AR:

kotlinx.coroutines.JobCancellationException: Parent job is Cancelling; job=JobImpl{Cancelled}@eb5ab92
	at kotlinx.coroutines.JobSupport.getChildJobCancellationCause(JobSupport.kt:711)
	at kotlinx.coroutines.JobSupport.createCauseException(JobSupport.kt:717)
	at kotlinx.coroutines.JobSupport.cancelMakeCompleting(JobSupport.kt:692)
	at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:664)
	at kotlinx.coroutines.JobSupport.parentCancelled(JobSupport.kt:634)
	at kotlinx.coroutines.ChildHandleNode.invoke(JobSupport.kt:1465)
	at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:1500)
	at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:897)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:860)
	at kotlinx.coroutines.JobSupport.cancelMakeCompleting(JobSupport.kt:693)
	at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:664)
	at kotlinx.coroutines.JobSupport.childCancelled(JobSupport.kt:648)
	at kotlinx.coroutines.ChildHandleNode.childCancelled(JobSupport.kt:1466)
	at kotlinx.coroutines.JobSupport.cancelParent(JobSupport.kt:358)
	at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
	at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:897)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:860)
	at kotlinx.coroutines.JobSupport.makeCompleting$kotlinx_coroutines_core(JobSupport.kt:803)
	at kotlinx.coroutines.JobImpl.completeExceptionally(JobSupport.kt:1326)
	at io.ktor.client.engine.apache.ApacheRequestProducer.failed(ApacheRequestProducer.kt:72)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:98)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:276)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	... 1 more
Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 178; received: 0)
	at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:89)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:52)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:18)
	at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1631)
	at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
	at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:51)
	at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	... 10 more

ER: ConnectionClosedException is not masked

org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 178; received: 0)
	at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:89)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:52)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:18)
	at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1631)
	at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
	at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:51)
	at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)

The list contains TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 here : https://github.com/ktorio/ktor/blob/master/ktor-network/ktor-network-tls/common/src/io/ktor/network/tls/CipherSuites.kt#L144-L152

but it is ignored by isSupported method here : https://github.com/ktorio/ktor/blob/master/ktor-network/ktor-network-tls/jvm/src/io/ktor/network/tls/CipherSuitesJvm.kt#L9-L12

which makes these cipher suites unavailable to be used because of : https://github.com/ktorio/ktor/blob/master/ktor-network/ktor-network-tls/jvm/src/io/ktor/network/tls/TLSHandshakeType.kt#L70

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/893

When configuring a very very long session duration it may cause integer overflow. It need to be decided what should be done in this case:

• fail at attempt to set duration
• bump max-age to Int.MAX_VALUE or Long.MAX_VALUE (long type requires changes, probably breaking changes)

This is related to #892.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1950

Ktor Version and Engine Used (client or server and name) https://ktor.io/quickstart/quickstart/gradle.html#intellij-improve-the-app-with-the-application-object

Describe the bug Ktor is currently at version 1.3.2, but in the quickstart docs it is mentioned that is not yet 1.0. Referenced line is: "Since Ktor is not yet 1.0, we have custom Maven repositories for distributing our early preview artifacts."

Expected behavior Line should be corrected.

https://ktor.io/docs/docker.html

The example does not work and is outdated.

It seems that the search functionality doesn't work in the docs.

Chrome Version 88.0.4324.192, macOS.

Ktor plugin generates a project with Kotlin 1.4.10, but the latest version is 1.4.31

Occurred 10 times in the last 5 or so minutes, just doing basic scrolling and picking and choosing of features.

java.lang.NoSuchMethodError: 'void org.intellij.plugins.markdown.ui.preview.MarkdownHtmlPanel.render()'
	at io.ktor.initializr.intellij.features.KtorFeatureDetailsPageComponent$showKtorFeature$3$1.run(KtorFeatureDetailsPageComponent.kt:263)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:776)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:746)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:969)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:839)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:449)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:808)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:448)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:781)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:502)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

IntelliJ IDEA 2021.1 Beta (Ultimate Edition)
Build #IU-211.6305.21, built on March 4, 2021
IntelliJ IDEA EAP User
Expiration date: April 4, 2021
Runtime version: 11.0.10+9-b1341.18 x86_64
VM: Dynamic Code Evolution 64-Bit Server VM by JetBrains s.r.o.
macOS 10.14.6
GC: G1 Young Generation, G1 Old Generation
Memory: 4096M
Cores: 8
Registry: ide.images.show.chessboard=true
Non-Bundled Plugins: com.jetbrains.darkPurpleTheme (1.2), org.nik.presentation-assistant (1.0.9), wu.seal.tool.jsontokotlin (3.6.1), org.asciidoctor.intellij.asciidoc (0.32.26), stardust (1.5.2105), com.jetbrains.intellij.api.watcher (6.46.0), intellij.ktor (1.5.1-eap-1)
Kotlin: 211-1.4.21-release-IJ6305.1

In ktor-samples/client-mpp Android Studio doesn't see imported dependencies in the iosMain sourceSet:

MacOS 11.1 Android Studio 4.1.1 XCode 12.3

Error message when trying to build the project iOS configuration of ktor-samples/client-mpp in Android Studio:

/Users/lukaszkalnik/projects/learning/ktor-samples/client-mpp/client-mpp-ios/ViewController.swift:3:8: error: no such module 'client_mpp'
import client_mpp
       ^

MacOS 11.1 Android Studio 4.1.1 XCode 12.3

Android Studio logs attached.

The route handler for /res/{parameter} will now also match for /res/. Before /res and /res/ would match the "getAll" resource. This means any "trailing" path segment parameter is implicitly treated as optional. So all handlers need to check if the parameter is empty string and then either

1. redirect to /res
2. copy&paste the handler of /res to the empty string code branch
3. give a 404 or 400

I guess "fixing" the behavior of /res == /res/ in 1.5 is opinionated, so there should be a real option/flag to ignore trailing slashes. Regression starting from 1.5 ( https://youtrack.jetbrains.com/issue/KTOR-372 ), works as expected in 1.4.3

The docs seem to be partially broken now for mandatory path segment parameters.

matches a path segment and captures it as a parameter named param. This path segment is mandatory, but you can make it optional by adding a question mark: {param?}.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1959

CIO Server ktor-server-cio:1.3.2-1.4-M2

Describe the bug CIO server always start on "0.0.0.0" - does not respect "connector" configuration

To Reproduce Steps to reproduce the behavior:

embeddedServer(CIO, applicationEngineEnvironment {
            module { ... }
            connector {
                host = "192.168.0.1"
                port = 8080
            }
})

Server will start on "0.0.0.0:8080" instead of "192.168.0.1:8080"

Expected behavior Server respect connector configuration.

I got this random error out of nowhere. I'm not even using the Ktor server. It's just random bots on the web making HTTP calls to it. I have no idea what request, call or response it was.

2021-01-19 21:21:41.833 E Unhandled exception caught for CoroutineName(call-handler)
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
	at io.netty.handler.codec.http.DefaultFullHttpRequest.release(DefaultFullHttpRequest.java:103)
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
	at io.ktor.server.netty.NettyApplicationCall.finishComplete(NettyApplicationCall.kt:53)
	at io.ktor.server.netty.NettyApplicationCall.finishSuspend(NettyApplicationCall.kt:46)
	at io.ktor.server.netty.NettyApplicationCall$finishSuspend$1.invokeSuspend(NettyApplicationCall.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:227)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:832)

Installed features: CallLogging, Compression, DefaultHeaders, CORS, XForwardedHeaderSupport, WebSockets (unused)

Routes: 1x get, 1x post (both for GraphQL queries)

Imagine that you have feature X and install it. Now you want to set up the feature config for the specific subroute. In the current state, there is no generic way to do this. Users can change the config of the feature after install (ex, Authentication feature) but only globally, and to do this, they need to understand how internal systems work. Also, implementing a new DSL method in routing can break a priority of routes (see https://youtrack.jetbrains.com/issue/KTOR-613)

Suggested solution:

API

Not all the features can be dynamically configured in routing. For example, feature that starts a background process on install should not know anything about routing. To separate them in API we will create the new interface:

public interface RoutingScopedFeature<in TPipeline : Pipeline<*, ApplicationCall>, TConfiguration : Any, TFeature : Any> :
    ApplicationFeature<TPipeline, TConfiguration, TFeature> {

    ... 

    /**
     * Feature installation script
     */
    public fun install(pipeline: TPipeline): TFeature
}

Note that the install function doesn't have a configuration block as a parameter on the installation, because it can be modified in routing.

The new overload for the install function will be introduced

public fun <Config, Feature : RoutingScopedFeature<*, Config, *>> Route.install(
    feature: Feature,
    configBuilder: Config.() -> Unit
)

To receive actual configuration inside the feature, the user will need to use the configurationBlock property inside the interceptor.

pipeline.intercept(phase) {
    val configuration = Configuration().apply(configurationBlock)
    ...
}

Usage example (Authentication feature):

install(Authentication) {
    form { validate { c -> UserIdPrincipal(c.name) } }
}

routing {
    route(...) {
        install(Authentication) {
            form("2") {
                validate { it.name.takeIf { it == "bbb" }?.let { UserIdPrincipal(it) } }
            }
        }
     
        get("/") { }
    }
}   

Implementation

To check full (but draft) implementation with tests and migration of the Authentication feature, please check the rsinukov/feature-config branch.

Migration

We can gradually add dynamic configuration functionality to existing features by changing their base interface from ApplicationFeature to DynamicConfigFeature. The migration of each feature will require minimal efforts (can be seen in the rsinukov/feature-config branch for the Authentication feature).

To reproduce run the server using the following code snippet:

fun main() {
    embeddedServer(Netty, port = 8081) {
        install(Authentication) {
            digest("digest") {
                val password = "123"
                realm = "testrealm@host.com"
                digestProvider { name, realm ->
                    "$name:$realm:$password".toByteArray()
                }
            }
        }
        routing {
            authenticate("digest") {
                get("/digest") {
                    call.respondText("ok")
                }
            }
        }
    }.start(wait = true)
}

Send the following HTTP request using curl:

curl -v  'http://localhost:8081/digest' --digest -u guest:123

I expect successfully pass authentication and get a response with a status code 200 but instead, I get the following response:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="43f48979ca35e2a5", algorithm="MD5"
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8081/digest'
* Found bundle for host localhost: 0x7fbe62518750 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> Authorization: Digest username="guest", realm="testrealm@host.com", nonce="43f48979ca35e2a5", uri="/digest", response="ee8c56f444996c77cd1e5f7570035372", algorithm="MD5"
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="76d13a440ec99b1c", algorithm="MD5"
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

The list allCompileKotlinTasks was removed due to refactoring

Need to install missing dependencies:

libncurses5 libncursesw5 libtinfo5

The project doesn't compile

Fix tests failing due to compiler error. The following error is in references

https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreMacOSJava11/21952 https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreWindowsJava11/21949 https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreMacOSJava8/21953 https://ktor.beta.teamcity.com/buildConfiguration/Ktor_KtorMatrixCoreWindowsJava8/21951

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1936

Ktor 1.3.2, TestApplicationEngine

When running a test that sends post with multipart it runs forever and never stops.

No problem when server really runs. It's just the test.

To Reproduce

Server:

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)

@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(ContentNegotiation) {
    }

    routing {
        post("/") {
            val mp = try {
                call.receiveMultipart()
            } catch (e: Exception) {
                call.application.log.error("Getting multipart error"")
                null
            }

           if (mp == null) call.respond(HttpStatusCode.BadRequest, "Null value")
           else call.respond(HttpStatusCode.OK,"OK")
        }
    }
}

Test:

internal class ApplicationKtTest {
    private val boundary = "***bbb***"
    private val  sysTempDir = System.getProperty("java.io.tmpdir") ?: "/tmp"

    private val multipart = listOf(PartData.FileItem({ byteArrayOf(1, 2, 3).inputStream().asInput() }, {}, headersOf(
            HttpHeaders.ContentDisposition,
            ContentDisposition.File
                    .withParameter(ContentDisposition.Parameters.Name, "file")
                    .withParameter(ContentDisposition.Parameters.FileName, "test.jpg")
                    .toString()
    )))

    @Test
    fun testUploadApplication() = testApp {
        handlePost("/", boundary, multipart).apply {
            println("============= RESPONSE ====================")
            println(response.content)
            println("=================================")
        }
    }
}

private fun TestApplicationEngine.handlePost(uri: String,
         boundary: String,
         multipart: List<PartData.FileItem>,
         setup: TestApplicationRequest.() -> Unit = {}
): TestApplicationCall {
    return handleRequest(method = HttpMethod.Post, uri = uri)  {
        addHeader(HttpHeaders.ContentType,
                ContentType.MultiPart.FormData.withParameter("boundary", boundary).toString()
        )
        setBody(boundary, multipart)
        setup()
    }
}

private fun testApp(callback: TestApplicationEngine.() -> Unit): Unit {
    withTestApplication({ module(true) }, callback)
}

The test runs forever and gives no result. if addHeader and setBody are commented there is a response.

Expected behavior The test should print OK

Repository If you just want to clone a test repository.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1040

Ktor Version

1.1.3

Ktor Engine Used(client or server and name)

Client iOS with default engine

Feedback (Bug)

When adding Firebase Performance to an iOS application, the app crashes when making a request with Ktor.

I've forked the sample mmp project and added the Firebase Performance library. The application crashes due to Firebase Performance. It might be because Firebase intercepts all network requests and it might be incompatible with Ktor.

You can find the sample mmp client crashing with Firebase Performance here https://github.com/eduardbosch/ktor-samples/commit/35bb435b4138ef966eeb9c44cbaf632bfaaeb305 Remember to add the GoogleServices-Info.plist to prevent Firebase to crash when it is initialised.

I hope this can help Ktor! 🎉

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1095

Ktor Version

1.1.5

Ktor Engine Used(client or server and name)

CIO server

JVM Version, Operating System and Relevant Context

1.8.0_131

Feedback

When using the CIO server engine, requests with more than 32 headers are rejected with error 400 and no further diagnostic. I used the debugger to further track the cause of this error and found that during header parsing an exception is thrown:

kotlin.NotImplementedError: An operation is not implemented: Implement headers overflow

This is due to the following declaration in file HttpHeaderMap.kt:

package io.ktor.http.cio

import io.ktor.http.cio.internals.* import io.ktor.util.* import kotlinx.io.pool.*

private const val EXPECTED_HEADERS_QTY = 32

which effectively limits the number of headers to 32. This error does not appear with the Jetty engine.

It's deprecated and looks like TextEncoder and TextDecoder work on majority of environments.

Sometimes a log entry may appear at the "wrong" place (before or after expected) so the testLogResponseWithException and siblings are flaky.

CIO native selector doesn't select new descriptors and unable to stop when closing because cancelled interests are not processed as well.

Here's how I noticed:

When I cancel the coroutine containing the WebSocket connection, KTor closes the connection with a 1011. If the server then responds by closing the connection without a status code, KTor will get a 1005 and it'll try to pass it to OkHTTP.

However, 1005 is a reserved status, so OkHTTP refuses it when Ktor passes it in the following lines: https://github.com/ktorio/ktor/blob/6053fa44a61fdfc5474ff909c0224e56877b5da1/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L62

Here's an example of a traceback you'd get:

    java.util.concurrent.CancellationException: ActorCoroutine was cancelled
        at kotlinx.coroutines.ExceptionsKt.CancellationException(Exceptions.kt:22)
        at kotlinx.coroutines.channels.ActorCoroutine.onCancelling(Actor.kt:134)
        at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
        at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:916)
        at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:875)
        at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:840)
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
     Caused by: java.lang.IllegalArgumentException: Code 1005 is reserved and may not be used.
        at okhttp3.internal.ws.WebSocketProtocol.validateCloseCode(WebSocketProtocol.kt:134)
        at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:435)
        at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:427)
        at io.ktor.client.engine.okhttp.OkHttpWebsocketSession$outgoing$1.invokeSuspend(OkHttpWebsocketSession.kt:62)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) 

I think there are technically two different issues happening at the same time:

• Ktor is trying to close the connection twice. The first time with a 1011 status (which I can confirm in the server logs) and the second time with a 1005 (because the server didn't provide a closing status code).
• Ktor is trying to close the connection with a reserved status code.

LoggingTest.testRequestContentTypeInLog and LoggingTest.testRequestAndResponseBody are fluky with Java client.

Quick investigation after fixing KTOR-1598 showed that the client dispatcher gets shutdown too early so the response body handling coroutines was unable to resume due to RejectedExecutionException.

The reason is that coroutines jobs structure was wrong and the client infrastructure was waiting for the wrong job instance.

val channel = HttpClient {
    Logging {
    }
}.get<ByteReadChannel>("/test")

channel.isClosedForRead == true
// channel is always closed with exception here

Tests HttpTimeoutTest.testConnectTimeout and testConnectTimeoutPerRequestAttributes somtimes may fail because of the unexpected exception text that is heavily system-dependant.

This is not a test issue but a real bug in okhttp integration

Tests testSocketTimeoutWriteFail* sometimes may pass because of missing failures on Apache. The possible reason is that sleep duration is only one second while timeouts are checked by apache every selection (that is also 1sec by default). So if the test server is too fast or selection is a little bit slower, we may get a response too early so no timeout happens in this case.

To reproduce run the following code:

fun main() {
    val client = HttpClient(Jetty) {}
    runBlocking {
        val r = client.get<String>("http://www.google.com")
        println(r)
    }
}

As a result, org.eclipse.jetty.io.EofException will be thrown:

Exception in thread "main" org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
	at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:259)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
	at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:543)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$jettyRequest$1.invoke(JettyHttpRequest.kt:40)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$jettyRequest$1.invoke(JettyHttpRequest.kt)
	at io.ktor.client.engine.jetty.UtilsKt.withPromise(utils.kt:14)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt.executeRequest(JettyHttpRequest.kt:39)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$1.invokeSuspend(JettyHttpRequest.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.nio.channels.AsynchronousCloseException
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
	... 18 more

If I change the URL to http://example.com then the app will hang indefinitely, here is a stacktrace:

"main" #1 prio=5 os_prio=0 cpu=337.00ms elapsed=56.16s tid=0x00007f8e98018800 nid=0x79547 waiting on condition  [0x00007f8e9eb77000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x00000006890f1518> (a kotlinx.coroutines.BlockingCoroutine)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:87)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at MainKt.main(main.kt:86)
	at MainKt.main(main.kt)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.36ms elapsed=56.11s tid=0x00007f8e98311000 nid=0x79550 waiting on condition  [0x00007f8e80382000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.7/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.7/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.7/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=1.63ms elapsed=56.11s tid=0x00007f8e98315000 nid=0x79551 in Object.wait()  [0x00007f8e80281000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.7/Native Method)
	- waiting on <0x00000006890f2008> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000006890f2008> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.7/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.22ms elapsed=56.10s tid=0x00007f8e9832a000 nid=0x79552 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=497.80ms elapsed=56.10s tid=0x00007f8e9832c000 nid=0x79553 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=306.59ms elapsed=56.10s tid=0x00007f8e9832e000 nid=0x79554 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #7 daemon prio=9 os_prio=0 cpu=6.70ms elapsed=56.09s tid=0x00007f8e98330000 nid=0x79555 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #8 daemon prio=8 os_prio=0 cpu=0.37ms elapsed=56.03s tid=0x00007f8e98381800 nid=0x79556 in Object.wait()  [0x00007f8e47efd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.7/Native Method)
	- waiting on <0x00000006890f27e0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000006890f27e0> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.7/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.7/InnocuousThread.java:134)

"Monitor Ctrl-Break" #9 daemon prio=5 os_prio=0 cpu=8.39ms elapsed=55.91s tid=0x00007f8e98695800 nid=0x79557 runnable  [0x00007f8e479d4000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.7/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.7/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.7/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.7/SocketInputStream.java:140)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.7/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.7/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(java.base@11.0.7/StreamDecoder.java:178)
	- locked <0x00000006890f4d00> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@11.0.7/InputStreamReader.java:185)
	at java.io.BufferedReader.fill(java.base@11.0.7/BufferedReader.java:161)
	at java.io.BufferedReader.readLine(java.base@11.0.7/BufferedReader.java:326)
	- locked <0x00000006890f4d00> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@11.0.7/BufferedReader.java:392)
	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:61)

"Service Thread" #10 daemon prio=9 os_prio=0 cpu=0.04ms elapsed=55.91s tid=0x00007f8e98696800 nid=0x79558 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"ktor-jetty-dispatcher-worker-1" #11 daemon prio=5 os_prio=0 cpu=230.54ms elapsed=55.68s tid=0x00007f8e989de000 nid=0x7955a waiting on condition  [0x00007f8e46ec0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:357)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

"ktor-jetty-dispatcher-worker-2" #12 daemon prio=5 os_prio=0 cpu=9.27ms elapsed=55.68s tid=0x00007f8e989df000 nid=0x7955b waiting on condition  [0x00007f8e46dbf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:357)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

"ktor-jetty-client-qtp-13" #13 prio=5 os_prio=0 cpu=6.95ms elapsed=55.47s tid=0x00007f8e340da000 nid=0x7955d runnable  [0x00007f8e46575000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.7/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.7/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.7/SelectorImpl.java:124)
	- locked <0x000000069f414300> (a sun.nio.ch.Util$2)
	- locked <0x000000069f4142a8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.7/SelectorImpl.java:141)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:472)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:409)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.io.ManagedSelector$$Lambda$65/0x0000000800176c40.run(Unknown Source)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-14" #14 prio=5 os_prio=0 cpu=126.50ms elapsed=55.47s tid=0x00007f8e340dc000 nid=0x7955e waiting on condition  [0x00007f8e46474000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-15" #15 prio=5 os_prio=0 cpu=0.27ms elapsed=55.47s tid=0x00007f8e340dd800 nid=0x7955f waiting on condition  [0x00007f8e46373000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069fc076c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2211)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:309)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:379)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-16" #16 prio=5 os_prio=0 cpu=2.68ms elapsed=55.46s tid=0x00007f8e340df800 nid=0x79560 waiting on condition  [0x00007f8e46272000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-17" #17 prio=5 os_prio=0 cpu=0.10ms elapsed=55.46s tid=0x00007f8e340e1000 nid=0x79561 waiting on condition  [0x00007f8e46171000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-18" #18 prio=5 os_prio=0 cpu=0.07ms elapsed=55.46s tid=0x00007f8e340e3000 nid=0x79562 waiting on condition  [0x00007f8e46070000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-19" #19 prio=5 os_prio=0 cpu=0.07ms elapsed=55.46s tid=0x00007f8e340e5000 nid=0x79563 waiting on condition  [0x00007f8e45f6f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-20" #20 prio=5 os_prio=0 cpu=0.06ms elapsed=55.46s tid=0x00007f8e340e7000 nid=0x79564 waiting on condition  [0x00007f8e45e6e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"Scheduler-779376001-1" #21 prio=5 os_prio=0 cpu=10.86ms elapsed=55.42s tid=0x00007f8e34117000 nid=0x79565 waiting on condition  [0x00007f8e45b6d000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f415c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.7/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.7/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.7/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.7/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"Attach Listener" #22 daemon prio=9 os_prio=0 cpu=8.56ms elapsed=55.08s tid=0x00007f8e4805c800 nid=0x79567 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=16.20ms elapsed=56.11s tid=0x00007f8e98309000 nid=0x7954f runnable  

"GC Thread#0" os_prio=0 cpu=22.88ms elapsed=56.16s tid=0x00007f8e98041800 nid=0x7954a runnable  

"GC Thread#1" os_prio=0 cpu=22.20ms elapsed=55.66s tid=0x00007f8e5c001000 nid=0x7955c runnable  

"G1 Main Marker" os_prio=0 cpu=0.51ms elapsed=56.15s tid=0x00007f8e980cc800 nid=0x7954b runnable  

"G1 Conc#0" os_prio=0 cpu=0.07ms elapsed=56.14s tid=0x00007f8e980ce000 nid=0x7954c runnable  

"G1 Refine#0" os_prio=0 cpu=0.50ms elapsed=56.13s tid=0x00007f8e9823b800 nid=0x7954d runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=10.66ms elapsed=56.13s tid=0x00007f8e9823d800 nid=0x7954e runnable  
"VM Periodic Task Thread" os_prio=0 cpu=42.21ms elapsed=55.91s tid=0x00007f8e98698000 nid=0x79559 waiting on condition  

JNI global refs: 14, weak refs: 0

If I make the same requests with curl: curl --http2-prior-knowledge http://example.com then I get an understandable error:

curl: (16) Error in the HTTP2 framing layer

Java client tests may freeze from time to time on CI. Debugging showed up that it's a bug.

To reproduce run ./gradlew runDebugExecutableMacosX64 on the attached project.

As a result, I get unexpected Unfinished workers detected, 1 workers leaked! warning and the program finishes with 134 exit code. As a workaround, I can add Platform.isMemoryLeakCheckerActive = false line to prevent this behavior.

Here is a code example that after execution reports Unfinished workers detected, 100 workers leaked! warning:

fun main() {
    runBlocking {
        repeat(100) {
            val client = HttpClient(Curl)
            val data = client.get<String> {
                url("http://google.com")
            }
            println(data)
            client.close()
        }
    }
}

Previously a ResponseException could be serialized. See: https://github.com/ktorio/ktor/issues/1256. The Transient annotation was removed in a recent commit for the HttpResponse field. Now an exception is thrown when serializing a ResponseException.

Stack trace:

io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
java.io.NotSerializableException: io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
	at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
	at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
	at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
	at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
	at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
	at ExceptionsTest.serialize(ExceptionsTest.kt:26)
	at ExceptionsTest.testResponseExceptionSerializable(ExceptionsTest.kt:17)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)

Here is a test to reproduce the issue:

import io.ktor.client.call.*
import io.ktor.client.features.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.util.date.*
import io.ktor.utils.io.*
import java.io.*
import kotlin.coroutines.*
import kotlin.test.*

class ExceptionsTest {

    @Test
    fun testResponseExceptionSerializable() {
        val exception = createResponseException()

        val serialized = serialize(exception)
        val deserialized = deserialize(serialized)

        deserialized as ResponseException
    }

    private fun serialize(obj: Any): ByteArray {
        val b = ByteArrayOutputStream()
        val o = ObjectOutputStream(b)
        o.writeObject(obj)
        return b.toByteArray()
    }

    private fun deserialize(bytes: ByteArray): Any? {
        val b = ByteArrayInputStream(bytes)
        val o = ObjectInputStream(b)
        return o.readObject()
    }
}

private fun createResponseException(): ResponseException = ResponseException(object : HttpResponse() {
    override val call: HttpClientCall
        get() = TODO("Not yet implemented")
    override val status: HttpStatusCode
        get() = TODO("Not yet implemented")
    override val version: HttpProtocolVersion
        get() = TODO("Not yet implemented")
    override val requestTime: GMTDate
        get() = TODO("Not yet implemented")
    override val responseTime: GMTDate
        get() = TODO("Not yet implemented")
    override val content: ByteReadChannel
        get() = TODO("Not yet implemented")
    override val headers: Headers
        get() = TODO("Not yet implemented")
    override val coroutineContext: CoroutineContext
        get() = TODO("Not yet implemented")

    override fun toString(): String = "FakeCall"
}, cachedResponseText = "Fake text")

version: 1.4.3

when use a http proxy, get the exception.

use no proxy, it's ok.

code

val httpClient = HttpClient(CIO) {
    engine {
        proxy = httpProxy
    }
}

val c : String = httpClient.get(url)

build.gradle.kts

// ...

dependencies {
    implementation("org.jetbrains.kotlinx", "kotlinx-serialization-json", "1.0.1")
    implementation("io.ktor", "ktor-client-core", "1.4.3")
    implementation("io.ktor", "ktor-client-cio", "1.4.3")
    implementation("io.ktor", "ktor-client-serialization-jvm", "1.4.3")
    implementation("org.jetbrains.kotlinx", "kotlinx-coroutines-core", "1.4.2")

    // test
}

exception:

Channel was closed
kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
	at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
	at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
	at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
	at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
	at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
	at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
	(Coroutine boundary)
	at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)
	at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:76)
	at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
	(Coroutine creation stacktrace)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30)
	at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
	at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
	at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:128)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:92)
	at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:123)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at LoginTest$testLogin$1.invokeSuspend(LoginTest.kt:18)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at mytest(...)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:205)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:201)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:129)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
Caused by: kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
	at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
	at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
	at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
	at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
	at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
	at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

In production I've got two threads stuck in 100% CPU usage due to Ktor.

The server is running without debugging agent, so I cannot debug the exact scenario just yet. I'll restart it today with debugging agent running so that I can investigate further the next time the issue occurs.

When creating thread dumps I get one of the following two outputs. Note the high CPU usage and that the threads seem to be stuck in ByteBufferChannel.readRemainingSuspend.

"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51203112.18ms elapsed=53179.31s tid=0x00007f6fc0052510 nid=0xa9b2 runnable  [0x00007f6f0ebe9000]
   java.lang.Thread.State: RUNNABLE
	at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible$default(ByteBufferChannel.kt:2678)
	at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:2231)
	at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51253887.85ms elapsed=53230.30s tid=0x00007f6fc0052510 nid=0xa9b2 runnable  [0x00007f6f0ebe9000]
   java.lang.Thread.State: RUNNABLE
	at io.ktor.utils.io.core.internal.UnsafeKt.prepareWriteHead(Unsafe.kt:160)
	at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:3060)
	at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

I upgraded my project to ktor 1.4.1 today. I found an issue when I ran my unit test. My logger failed to log the SocketTimeoutException.

Sample code:

fun main() {
    val httpClient = HttpClient(Apache) {
        install(HttpTimeout) {
            socketTimeoutMillis = 3000
        }
    }

    runBlocking {
        try {
            val get = httpClient.get<String>("https://httpbin.org/delay/10")
        } catch (e: Exception) {
            e.printStackTrace()
            LoggerFactory.getLogger("main").error("test", e)
        }
    }
}

Output:

io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
	at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:49)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt$sendRequest$1.invokeSuspend(ApacheHttpRequest.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
	at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
	at io.ktor.client.engine.apache.ApacheResponseConsumer.failed(ApacheResponseConsumer.kt:73)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:99)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:834)
	[CIRCULAR REFERENCE:io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms]
Caused by: java.net.SocketTimeoutException: 3,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	... 11 more
Exception in thread "main" java.lang.StackOverflowError
	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:590)
	at java.base/java.lang.reflect.Method.invoke(Method.java:558)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:66)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:72)

I already made a pr on github: https://github.com/ktorio/ktor/pull/2237

Subsystem ktor-network-jvm

Motivation TCP has a well known self-connect problem, which client can connect to the client itself without any program listen on the port.

Reproduce Code :

fun testSelfConnect() {
    runBlocking {
        // Find a port that would be used as a local address.
        val port = getAvailablePort()

        val tcpSocketBuilder = aSocket(ActorSelectorManager(Dispatchers.IO)).tcp()
        // Try to connect to that address repeatedly.
        for (i in 0 until 100000) {
            try {
                val socket = tcpSocketBuilder.connect(InetSocketAddress("127.0.0.1", port))
                println("connect to self succeed: ${socket.localAddress} to ${socket.remoteAddress}")
                System.`in`.read()
                break
            } catch (ex: Exception) {
                // ignore
            }
        }
    }
}

// since new linux kernel version introduce a feature, new bind port number will always be odd number
// and connect port will always be even, so we find a random even port with while loop
// see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=07f4c90062f8fc7c8c26f8f95324cbe8fa3145a5
private fun getAvailablePort(): Int {
    while (true) {
        val port = ServerSocket().apply {
            bind(InetSocketAddress("127.0.0.1", 0))
            close()
        }.localPort

        if (port % 2 == 0) {
            return port
        }

        try {
            // try bind the next even port
            ServerSocket().apply {
                bind(InetSocketAddress("127.0.0.1", port + 1))
                close()
            }
            return port + 1
        } catch (ex: Exception) {
            // ignore
        }
    }
}

Running the above code whill result in a tcp connection with src addr equal to dst addr and src port equal to dst port.

If this happens, client will behave abnormal because it connect to the wrong server，server will bind fail on this machine.

Other programming language also have this problem, eg golang: https://golang.org/issue/2690

Fix this problem is easy, just check if the local and remote socket pair are equal.

StartTimeoutTest.testTimeoutCancelsWhenParentScopeCancels is flaky: 8 failures in almost 1000 builds

The reason is that the timeout coroutine may trigger too fast after changing the test clock but before the parent cancellation

If you use URLBuilder to create a URL, it incorrectly encodes the path. Per https://tools.ietf.org/html/rfc3986#section-2 3.3, a The path is terminated by the first question mark ("?") or number sign ("#") character, or by the end of the URI.. However the # symbol isn't encoded by the path function.

val rawPath = "id+test&test~test#test"
val urlString = url {
    path(rawPath)
}
// urlString = http://localhost/id+test&test~test#test

If you look at the source of the path function you will see it uses <string>.encodeURLQueryComponent() which is the incorrect function. If it used the <string>.encodeURLPath() the # would be correctly encoded as id+test&test~test%23test.

public fun path(components: List<String>): URLBuilder {
    encodedPath = components.joinToString("/", prefix = "/") { it.encodeURLQueryComponent() }
    return this
}

Exceptions thrown from the UDP sockets cannot be caught.

Here is a test that currently fails:

import io.ktor.network.selector.*
import io.ktor.network.sockets.*
import io.ktor.util.network.*
import io.ktor.utils.io.core.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import kotlin.io.use
import kotlin.test.*

class SocketExceptionIssue : CoroutineScope {
    private val testJob = Job()
    private val selector = ActorSelectorManager(Dispatchers.Default + testJob)

    override val coroutineContext: CoroutineContext
        get() = testJob

    @AfterTest
    fun tearDown() {
        testJob.cancel()
        selector.close()
    }

    @Test
    fun testBroadcastFails(): Unit = runBlocking {
        withTimeout(15000) {
            // The following code in the assertFails lambda should throw a SocketException.
            assertFails {
                aSocket(selector)
                    .udp()
                    .bind {
                        // Do not set broadcast to true so exception is thrown.
                    }
                    .use { socket ->
                        // Send a broadcast which will fail because broadcasts are not enabled on this socket.
                        socket.send(
                            Datagram(
                                packet = buildPacket { writeText("0123456789") },
                                address = NetworkAddress("255.255.255.255", 56700)
                            )
                        )
                    }
            }
        }
    }
}

When running this test the logs show the following:

> Task :ktor-network:jvmTest
Exception in thread "DefaultDispatcher-worker-1 @coroutine#3" java.net.SocketException: Permission denied
	at java.base/sun.nio.ch.DatagramChannelImpl.send0(Native Method)
	at java.base/sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:584)
	at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:546)
	at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:529)
	at io.ktor.network.sockets.DatagramSocketImpl.sendImpl(DatagramSocketImpl.kt:98)
	at io.ktor.network.sockets.DatagramSocketImpl$sender$1.invokeSuspend(DatagramSocketImpl.kt:35)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Expected an exception to be thrown, but was completed successfully.
java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
	at org.junit.Assert.fail(Assert.java:88)
	at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
	at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
	at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
	at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
	(Coroutine boundary)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
	(Coroutine creation stacktrace)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at io.ktor.network.sockets.tests.SocketExceptionIssue.testBroadcastFails(SocketExceptionIssue.kt:30)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
	at org.junit.Assert.fail(Assert.java:88)
	at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
	at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
	at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
	at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:81)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:56)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
	... 50 more

io.ktor.network.sockets.tests.SocketExceptionIssue[jvm] > testBroadcastFails[jvm] FAILED
    java.lang.AssertionError at SocketExceptionIssue.kt:30
        Caused by: java.lang.AssertionError at SocketExceptionIssue.kt:30
1 test completed, 1 failed

You can see the SocketException in the log and that the test (with assertFails) is unable to catch it.

The following issue looks related: https://youtrack.jetbrains.com/issue/KTOR-440

Here is a fork of Ktor with the test added so you can reproduce the issue: https://github.com/Thomas-Vos/ktor/tree/socket_exception_issue (see last commit for added test). The fork is based on the 1.5.0 branch.

Add following information to the CONTRIBUTING document:

• How to publish artifacts locally - Maven Local or includeBuild...
• How to set up a sample project for local development - Referencing Ktor locally
• How to debug Ktor in IDEA
• How to write integration tests - Ask Team for that.

And maybe some additional information that will make the life of a contributor easier.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/225

Generating files using Ktor 1.2.4 and Websocket HTTP Client Support dependency makes it so the project will not build. build.gradle file specifies compile "io.ktor:ktor-client-websocket:$ktor_version" which will not work since ktor-client-websocket does not have v1.2.4 in the repositories.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/185

https://ktor.io/clients/http-client/features/logging.html clearly mentions io.ktor:ktor-client-logging artifact, but after adding that artifact and copy-pasting the code, it still doesn't compile because it also needs io.ktor:ktor-client-logging-jvm for example.

After adding that, it compiles, but then I need the hidden knowledge of https://ktor.io/servers/logging.html#providers (which is Server docs) to add something like implementation("org.slf4j:slf4j-simple:1.7.26")

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/255

https://github.com/ktorio/ktorio.github.io/blob/master/servers/self-signed-certificate.md

The documentation reads

To create a self-signed certificate using Ktor, you have to call the generateCertificate function.

io.ktor.network.tls.certificates.generateCertificate(File("mycert.jks"))

With

    implementation "io.ktor:ktor-network-tls:$ktor_version"

where ktor_version is 1.3.0, there is no generateCertificate function defined in io.ktor.network.tls.certificates

import io.ktor.network.tls.certificates.generateCertificate is an invalid import.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1208

Ktor Version

1.2.2

Feedback

I'm using Jackson mapping in the Client (here: https://ktor.io/clients/http-client/features/json-feature.html), but there seems to be no way to configure the mapper. Normally, I would just include a custom-configured object mapper (the same way I do for server component), but here it's just a single-constructor class tha explicitly calls for the default mapper inside.

Can we get a configuration param for the object mapper? Alternatively, a builder to specify it.

For example, the old documentation page about Jackson feature has ARTIFACT block but the corresponding new documentation page has not.

https://ktor.io/docs/features-auth.html

https://ktor.io/docs/servers-features-websockets.html

Serialization for client section does not explain how to use it and it doesn't tell anything about required headers. https://ktor.io/clients/http-client/features/json-feature.html -- This is the page that describes the Json feature. It only teaches how to install the feature (and reader assumes that it will work out of the box right after installation, and any data will be automatically serialized which is not quite true because it also requires contentType to be set).

This implies that user can just write something like:

@Serializable
data class HelloWorld(val text: String = "Hello world")

val client = HttpClient(Apache) {
    install(JsonFeature) {
        serializer = KotlinxSerializer()
    }
}

client.post<Unit> {
    url("http://127.0.0.1:8080/")
    body = HelloWorld()
}

But this is not true:

Exception in thread "main" java.lang.IllegalStateException: Fail to serialize body. Content has type: class HelloWorld, but OutgoingContent expected.
If you expect serialized body, please check that you have installed the corresponding feature(like `Json`) and set `Content-Type` header.
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:79)
	at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:185)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at MainKt$main$1.invokeSuspend(main.kt:49)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at MainKt.main(main.kt:19)

It also requires content type (ex., contentType(ContentType.Application.Json)) to be set. This information is not discoverable and should be documented.

To reproduce add api "io.ktor:ktor-client-mock-native:$ktor_version" artifact, that is described on the Testing Http Client (MockEngine) docs page, to the Gradle build file. If ktor_version is 1.4.0 then the build fails with unexpected error:

Could not find io.ktor:ktor-client-mock-native:1.4.0.

The migration guide for 1.4.0 explains that behavior.

https://ktor.io/docs/authentication.htm

https://ktor.io/servers/self-signed-certificate.html says :

• This function is defined in the method io.ktor.network.tls.certificates.generateCertificate in the artifact io.ktor:ktor-network-tls:$ktor_version.

But it is wrong for 1.3.x Ktor versions (see https://github.com/ktorio/ktor/issues/1612). The workaround is to use "io.ktor:ktor-network-tls-certificates:$ktor_version" for 1.3.x. Samples also need to be fixed

when hitting https://ktor.io/docs/servers-testing.html getting 404

The fatjar documentation page https://ktor.io/docs/fatjar.html#fat-jar-gradle recommends using shadow plugin 2.0.4.

But I faced this error:

Task :shadowJar FAILED A problem was found with the configuration of task ':shadowJar' (type 'ShadowJar'). No value has been specified for property 'mainClassName'.

I fixed this by updating the plugin from 2.0.4 to 6.1.0 (latest)

Example project: https://github.com/egorklimov/ktor-fatjar-issue

Like we have on https://kotlinlang.org/docs/home.html, to encourage external contributors to provide documentation content and fixes.

The plugin allows using empty fields for project.name

I saw that I could add dropwizard-metrics to Ktor and get it to create metrics for resources and such, really neat! In my app I already have a metrics registry that I have added some metric sets for threads, memory and such to. When I add my existing registry to Ktor like this:

install(DropwizardMetrics) {
    registry = metricRegistry
}

When starting the app I realize that Ktor registers the same metric sets and it is causing issues. Like this: Failed to start WebServer, due to: 'A metric named jvm.memory.heap.committed already exists', retrying How do I register my existing metrics registry and avoid having Ktor register the same metrics again?

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1868

Ktor Version and Engine Used (client or server and name) io.ktor:ktor-server-cio:1.3.2

Describe the bug Missing header name should fail(see https://tools.ietf.org/html/rfc2616#section-4.2)

To Reproduce Steps to reproduce the behavior: Run code:

import io.ktor.http.cio.parseRequest
import io.ktor.utils.io.ByteReadChannel
import kotlinx.coroutines.runBlocking

fun main() {
    test(
        """
        GET / HTTP/1.1
        Host: www.example.com
        : value


        """.trimIndent()
    )
}

fun test(http: String) {
    val request = runBlocking { parseRequest(ByteReadChannel(http)) }!!
    println("method=${request.method};version=${request.version};uri={${request.uri}};headers=[${request.headers}]")
}

Expected behavior An error expected - exception or null result.

After introducing accepted headers predicate, CORS started to accept non-simple headers by default. It is not a security issue because clients will not pass it anyway in spite of success.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1882

Ktor Version and Engine Used (client or server and name) io.ktor:ktor-server-cio:1.3.2

Describe the bug Status-code must be 3-digit(see https://tools.ietf.org/html/rfc2616#section-6.1.1)

To Reproduce Steps to reproduce the behavior: Run code:

import io.ktor.http.cio.parseResponse
import io.ktor.utils.io.ByteReadChannel
import kotlinx.coroutines.runBlocking


fun main() {
    listOf(
        """
        HTTP/1.1 0 OK


        """.trimIndent(),
        """
        HTTP/1.1 10 OK


        """.trimIndent(),
        """
        HTTP/1.1 1000 OK


        """.trimIndent(),
        """
        HTTP/1.1 10000 OK


        """.trimIndent()
    ).forEach { test(it) }

}

fun test(http: String) {
    val response = runBlocking { parseResponse(ByteReadChannel(http)) }!!
    println("status=${response.status};statusText={${response.statusText}};headers=[${response.headers}];version=${response.version}")
}

Expected behavior An error expected - exception or null result.

If the input channel is broken due to some reason (broken connection or wrong transfer encoding parameters), the server may lose the exception and receive an empty request body and continue working.

Test CallLoggingTest.can log application lifecycle events crashes with the following message:

java.lang.IndexOutOfBoundsException: Index 14 out of bounds for length 6 at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) at java.base/java.util.Objects.checkIndex(Objects.java:372) at java.base/java.util.ArrayList.get(ArrayList.java:459) at io.ktor.tests.server.features.CallLoggingTest.can log application lifecycle events(CallLoggingTest.kt:64)

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1293

Ktor client version: 1.2.3

URLBuilder(host = "localhost").apply {
    path("?")
    parameters["?"] = "?"
}
    .buildString()
    .also(::println)

will produce:

http://localhost/??%3F=%3F

I would expect the following url:

http://localhost/%3F?%3F=%3F

URLBuilder uses encodeURLQueryComponent function to encode a path: https://github.com/ktorio/ktor/blob/1.2.3/ktor-http/common/src/io/ktor/http/URLBuilder.kt#L50 Possibly it should use encodeURLPath?

Not sure it is a bug, but a feature. Fixing it will break backward compatibility.

Ktor Version

1.4.2

Ktor Engine

Netty

JVM Version, Operating System and Relevant Context

JDK 11, Manjaro

Problem

1. Project is generated by Ktor IDEA plugin, which if test modules are included, generates a module with default parameters.
2. It fails with Module function provided as lambda cannot be unlinked for reload, if module has default parameters.
3. Using method reference as in the example, does not make it work.

Example:

fun main(args: Array<String>): Unit {
    embeddedServer(
            Netty,
            port = 8080,
            host = "0.0.0.0",
            watchPaths = listOf("some"),
            module = Application::module,
    ).start(true)
}

fun Application.module(testing: Boolean = false): Unit {
    routing {
        get("/") {
            call.respondText { "Hello" }
        }
    }
}

Running Result

Exception in thread "main" java.lang.RuntimeException: Module function provided as lambda cannot be unlinked for reload
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.<init>(ApplicationEngineEnvironmentReloading.kt:58)
	at io.ktor.server.engine.ApplicationEngineEnvironmentBuilder.build(ApplicationEngineEnvironment.kt:105)
	at io.ktor.server.engine.ApplicationEngineEnvironmentKt.applicationEngineEnvironment(ApplicationEngineEnvironment.kt:44)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:55)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:36)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:34)
	at some.ReloadKt.main(reload.kt:10)

It fails here, basically can't cast the default parameter method as KFunction<*>

public class ApplicationEngineEnvironmentReloading(
//...
) : ApplicationEngineEnvironment {
//...

    private val moduleFunctionNames: List<String>? = run {
        val configModules = config.propertyOrNull("ktor.application.modules")?.getList()
        if (watchPatterns.isEmpty()) configModules
        else {
            val unlinkedModules = modules.map {
         // Fails here
                val fn = (it as? KFunction<*>)?.javaMethod
                    ?: throw RuntimeException("Module function provided as lambda cannot be unlinked for reload")
                val clazz = fn.declaringClass
                val name = fn.name
                "${clazz.name}.$name"
            }
            if (configModules == null)
                unlinkedModules
            else
                configModules + unlinkedModules
        }
    //...
}

Workaround

fun Application.module() = module(false)

fun Application.module(testing: Boolean = false): Unit {...}

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1946

Subsystem Client, and server possibly

Is your feature request related to a problem? Please describe.

    client.ws(host = "echo.websocket.org") {
        send(Frame.Text("Hello World"))
        close()

        incoming.consumeEach {
            println(it)
        }
    }

Fails with:

Exception in thread "main" java.util.concurrent.CancellationException: ArrayChannel was cancelled
	at kotlinx.coroutines.channels.AbstractChannel.cancel(AbstractChannel.kt:632)
	at kotlinx.coroutines.channels.ReceiveChannel$DefaultImpls.cancel$default(Channel.kt:311)
	at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl$runOutgoingProcessor$1.invokeSuspend(DefaultWebSocketSessionImpl.kt:163)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)
	at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)
	at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
	at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:306)
	at kotlinx.coroutines.CancellableContinuationImpl.completeResume(CancellableContinuationImpl.kt:393)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveHasNext.completeResumeReceive(AbstractChannel.kt:918)
	at kotlinx.coroutines.channels.ArrayChannel.offerInternal(ArrayChannel.kt:83)
	at kotlinx.coroutines.channels.AbstractSendChannel.send(AbstractChannel.kt:149)
	at io.ktor.http.cio.websocket.WebSocketSession$DefaultImpls.send(WebSocketSessionJvm.kt:45)
	at io.ktor.http.cio.websocket.DefaultWebSocketSession$DefaultImpls.send(DefaultWebSocketSession.kt)
	at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl.send(DefaultWebSocketSessionImpl.kt:27)
	at io.ktor.client.features.websocket.DefaultClientWebSocketSession.send(ClientSessions.kt)
	at io.ktor.http.cio.websocket.WebSocketSessionKt.close(WebSocketSession.kt:74)
	at io.ktor.http.cio.websocket.WebSocketSessionKt.close$default(WebSocketSession.kt:72)
	at WebSocketKt$Websockets$1$1.invokeSuspend(WebSocket.kt:15)
	at WebSocketKt$Websockets$1$1.invoke(WebSocket.kt)
	at io.ktor.client.features.websocket.BuildersKt.webSocket(builders.kt:54)
	at io.ktor.client.features.websocket.BuildersKt$webSocket$1.invokeSuspend(builders.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:272)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at WebSocketKt.Websockets(WebSocket.kt:7)
	at MainKt.main(main.kt:4)

Some services require header params to be double-quoted. In contrast, we follow RFC that says that parameter can be non quoted if there are no special characters. To use Ktor client with such services, users add quotes to parameters manually. Sometimes it results in double-quotes.

When upgrading to 1.5.0 from 1.4.3 I'm getting exception kotlinx.serialization.SerializationException: Class 'ArrayList' is not registered for polymorphic serialization in the scope of 'Collection'

This test triggers the problem (works in 1.4.3):

class ArrayListSerializationBug {
    @Test fun testListSerialization() = withTestApplication({
        install(ContentNegotiation) {
            json()
        }
        routing {
            get("/list") {
                val response: Collection<String> = listOf("a", "b")
                call.respond(response)
            }
        }
    })
    {
        with(handleRequest(Get, "/list"), fun TestApplicationCall.() {
            assertEquals(OK, response.status())
            assertEquals("""["a","b"]""", response.content)
        })
    }
}

It results in unexpected rejections during decompression and logging:

java.util.concurrent.CancellationException: The task was rejected
	at kotlinx.coroutines.ExceptionsKt.CancellationException(Exceptions.kt:22)
	at kotlinx.coroutines.ExecutorCoroutineDispatcherBase.cancelJobOnRejection(Executors.kt:144)
	at kotlinx.coroutines.ExecutorCoroutineDispatcherBase.dispatch(Executors.kt:97)

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1764

Ktor Version 1.3.2 - Android

Post request shows empty body

We are sending a post request using Ktor. After upgrading to V1.3.2 to support kotlin 1.3.7, the request body started showing empty and nothing is actually sent in the body. I used the debugger to trace it, I see that it is able to serialize my request object correctly and content type is done correctly with application/json, however at some point I see in the logs that it is nullified.

We are using JsonFeature and Logging feature and our event object has it's own serilaizer which I was able to see it successfully serialized inside Ktor client

This is from KotlinxSerializer.kt

    override fun write(data: Any, contentType: ContentType): OutgoingContent {
        @Suppress("UNCHECKED_CAST")
        val content = json.stringify(buildSerializer(data) as KSerializer<Any>, data)
        return TextContent(content, contentType)
    }

It is returning a valid TextContent(content, contentType) with correct contentType and a valid json string as content

This is how I install the features

private val httpClient by lazy {
        HttpClient {
            install(Logging) {
                logger = apiLogger
                level = LogLevel.ALL
            }
            install(JsonFeature) {
                serializer = KotlinxSerializer()
            }
        }
    }

This is how we do the post request inside the same class

    fun fireEvents(event: List<EventRequest>) = blockingGet {
        httpClient.post<Unit>(
            scheme = SCHEME_HTTPS,
            host = baseUrl,
            path = EVENT_API_V1,
            body = event
        ) {
            header(CONTENT_TYPE, APPLICATION_JSON)
            header(API_TOKEN_HEADER, applicationApiToken)
        }
    }

In previous version I used to see the body correctly. But now I see the log just BODY START and BODY END

    METHOD: HttpMethod(value=POST)
    COMMON HEADERS
    -> some-key: some-value
    -> Accept: application/json
    -> Accept-Charset: UTF-8
    CONTENT HEADERS
    BODY Content-Type: application/json
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
I/AppLogger: RESPONSE: 200 OK
    METHOD: HttpMethod(value=POST)
I/AppLogger: FROM: https://something.com/v1/send
    COMMON HEADERS
    -> Connection: keep-alive
    -> Date: Tue, 31 Mar 2020 14:58:04 GMT
    -> Server: nginx/1.14.1
    -> Transfer-Encoding: chunked
    -> X-Android-Received-Millis: 1585666684496
    -> X-Android-Response-Source: NETWORK 200
    -> X-Android-Selected-Protocol: http/1.1
    -> X-Android-Sent-Millis: 1585666684232
I/AppLogger: BODY Content-Type: null
    BODY START
I/AppLogger: BODY END

I am not sure if this is normal or not. But can you tell me if I am doing something wrong in my code that I can fix? I checked changelog but I did not see anything that requires my attention in this code snippet I posted.

route("/resource") {
    get("/") {
        ...
    }
}

The above code has the same behavior as this:

route("/resource") {
    get("") {
        ...
    }
}

The expected behavior should be the same as this:

route("/resource/") {
    get("") {
        ...
    }
}

Ktor 1.5 requires explicitly handling trailing slashes or not, which works in general, for example in the last code snippet here. The "/resource/" is properly parsed to create the equivalent route. But if we "split" the last slash into a nested route like in the first code example, the last slash gets ignored and it is instead treated like the second code snippet. Noted that this only affects the nested routes with no additional path after the slash, a nested get("/abc") still works.

Expected behavior: Trailing slash in a nested route is properly concatenated to the route Actual behavior: HTTP 404 Not found when calling the route with a trailing slash.

it would be useful to have the capability to highlight breaking changes somehow. For example, here we can filter breaking changes, see them for specific versions, and even set the version interval: https://supportcenter.devexpress.com/versionhistory AFAIU, it requires adding the 'isBreakingChange' field to Yourtrack (or a dedicated tag)

Provide/publish internal option to skip TLS verification for curl client engine

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1930

Subsystem Client, JVM

Describe the solution you'd like In Java 11 was introduced HTTP Client based on reactive-streams.

Motivation to include to ktor Since more projects are switching to modern Java versions it could be valuable to use built-in Java HTTP Client without need to use additional libraries. It supports HTTP/1.1 and HTTP/2 protocol and provides async APIs.

Test to reproduce below. Tested on two different machines (Windows and macOS), works fine with CIO. With 500 test iterations, happens at least once most times.

import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.apache.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.http.*
import io.ktor.http.content.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.cio.*
import io.ktor.server.engine.*
import io.ktor.utils.io.*
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.RepeatedTest
import java.util.concurrent.TimeUnit

class KtorApacheBugTest {

    private val port = 8000
    private val server = embeddedServer(
        factory = CIO,
        port = port,
    ) {
        routing {
            put("/") {
                call.receiveChannel().discard()
                call.respond(HttpStatusCode.OK)
            }
        }
    }
    private val client = HttpClient(Apache) {
        install(HttpTimeout) {
            socketTimeoutMillis = 10_000
            connectTimeoutMillis = 10_000
        }
    }

    @BeforeEach
    fun setUp() {
        server.start()
    }

    @AfterEach
    fun tearDown() {
        client.close()
        server.stop(0, 1, TimeUnit.SECONDS)
    }

    @RepeatedTest(500)
    fun `eventually hits a socket timeout`(): Unit = runBlocking {
        val chunk = (0 until 10_000).map { it.toByte() }.toByteArray()

        client.put<String>("http://localhost:${port}") {
            body = ChannelWriterContent(
                body = {
                    writeFully(chunk)
                },
                contentType = null,
            )
        }
    }
}

Another simple test (seems the issue is with ChannelWriterContent) below.

This works:

@RepeatedTest(500)
    fun `does not hit a socket timeout`(): Unit = runBlocking {
        client.put<String>("http://localhost:${port}") {
            body = LocalFileContent(File("gradle/wrapper/gradle-wrapper.jar"))
        }
    }

This does not:

@RepeatedTest(500)
    fun `eventually hits a socket timeout`(): Unit = runBlocking {
        val chunk = File("gradle/wrapper/gradle-wrapper.jar").readBytes()

        client.put<String>("http://localhost:${port}") {
            body = ChannelWriterContent(
                body = {
                    writeFully(chunk)
                },
                contentType = null,
            )
        }
    }

Sending the same via ByteArrayContent works fine as well.

Currently, Ktor calls OkHttpClient#newWebSocket to create an OkHttp WebSocket: https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L53

I would like to be able to provide my own instance of OkHttp WebSocket.Factory, so that I may override behaviour. Details of my use case are described here: https://github.com/square/okhttp/issues/6242

How do you feel about this? Thanks.

The following code fails with a ConnectTimeoutException. Maybe the property connectRetryAttempts is mislabeled and actually means the total number of attempts, not the number of retry attempts.

suspend fun main(): Unit = withContext(Dispatchers.Default) {
    HttpClient(CIO) {
        engine {
            endpoint.connectRetryAttempts = 0
        }
    }.use { client ->
        client.get<String>("https://www.google.com")
    }
}

Exception in thread "main" io.ktor.network.sockets.ConnectTimeoutException: Connect timeout has been expired [url=https://www.google.com/, connect_timeout=unknown ms]
	at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException(HttpTimeout.kt:164)
	at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException$default(HttpTimeout.kt:163)
	at io.ktor.client.engine.cio.Endpoint.getTimeoutException(Endpoint.kt:208)
	at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:200)
	at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Make the message of ResponseException contain response's text from response.readText().

When doing get to e.g. only inspect the etag header or some other response header, the status code isn't validated. It's only validated when accessing the body. However, we often don't inspect the body, but only the header. This inconsistency has caused bugs on our side because we missed bad responses.

Could you please always validate the response status code, no matter if the body is accessed or not?

Steps:

1. privoxy setup:

$ cat  ./privoxyconfig 
confdir /etc/privoxy
temporary-directory /tmp
logdir /var/log/privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
filterfile user.filter      # User customizations
logfile privoxy.log
debug     1 # Log the destination for each request Privoxy let through. See also debug 1024.
debug  1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
debug  4096 # Startup banner and warnings
debug  8192 # Non-fatal errors
listen-address  0.0.0.0:8118
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 1
buffer-limit 4096
enable-proxy-authentication-forwarding 0
trusted-cgi-referer http://www.example.org/
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
tolerate-pipelining 1
socket-timeout 300

$ cat ./Dockerfile 
FROM alpine
RUN apk add --no-cache privoxy=3.0.28-r0
COPY privoxyconfig /etc/privoxy/config
EXPOSE 8118
CMD privoxy --no-daemon /etc/privoxy/config

$ docker build . -t privoxy
$ docker run --rm -p 127.0.0.1:8118:8118 privoxy

2. set proxy as described in https://ktor.io/docs/proxy.html with

fun main(args: Array<String>) {
    val client = HttpClient(CIO) {
        install(UserAgent) {
            agent = "Space Unfurl Bot"
        }

        install(ContentEncoding) {
            gzip()
        }

        engine {
            proxy = Proxy(Proxy.Type.HTTP, InetSocketAddress("127.0.0.1", 8118))
        }
    }
    val r = runBlocking(CoroutineName("Get Metadata")) {
        client.get<HttpStatement>("https://example.com").execute { call ->
            "response code: ${call.status.description}"

        }
    }
    println(r)
}

Request: [X] https://example.com CIO hangs for a long time and returns java.lang.IllegalArgumentException: Invalid TLS record type code: 72 [X] http://ya.ru same result (ya.ru redirects to https) [V] http://example.com returns page content [V] Replace CIO with Apache - any requests work perfectly fine.

Problem

Sometimes, we introduce new API but, due to our compatibility guarantees, we can't make it public and forced to introduce it as internal, sometimes with magic to invoke such internals from other modules. Usually, it is intended to make them public at some point in the future, however, because of long minor/major release cycles, it's easy to forget to promote such APIs.

Proposed solution

The solution is to provide an annotation with the following signature

/**
 * API marked with this annotation is intended to become public in the future [version].
 * Usually it means that the API can't be public at the moment of development due to
 * compatibility guarantees restrictions.
 *
 * Marking a public declaration with this annotation makes no sense
 * except for the case when it is also marked with [InternalAPI].

 * Please note that the specified [version] and the fact of making something a candidate is not a guarantee, 
 * so the target version could be changed without any notice or even the promotion could be cancelled at all.
 * 
 * @property version in which the API is planned to be promoted
 */
@InternalAPI // so users shouldn't use this annotation, it's for ktor only
@Retention(AnnotationRetention.SOURCE)
@Target(
    AnnotationTarget.CLASS,
    AnnotationTarget.ANNOTATION_CLASS,
    AnnotationTarget.CONSTRUCTOR,
    AnnotationTarget.FUNCTION,
    AnnotationTarget.PROPERTY,
    AnnotationTarget.PROPERTY_SETTER,
    AnnotationTarget.TYPEALIAS
)
public annotation class PublicAPICandidate(val version: String)

Using this annotation, one could mark some internal API, specifying the desired future version. It is recommended to document such API in spite of that it is internal yet.

When a minor/major version starts (or in the middle of development period), one should search for all annotation usages and collect/notify developers who wanted to make something public. At the end of the version development, we should ensure that there are no annotations with this version remaining.

Every promoted API should be properly documented and have adequate tests.

Example

In 1.4.1 we introduce a property

// In 1.4.1
public class SomeConfig {
    /** Initial documentation */
    @PublicAPICandidate("1.5.0") // the next minor version
    internal var usefulThing: String = ""
}

So no breaking changes, just the internal property.

When we develop 1.5.0 we search for all annotation usages and promote it removing annotation.

// In 1.5.0
public class SomeConfig {
    /**
     * Documentation
     */
    public var usefulProperty: String = ""
}

The other good effect is that by doing this we declare our intention to make something public and declare which version is it. So ktor users may see it, and this avoids unnecessary quesions and improves clarity.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/272

I tried the kotlinx serialization example from the docs: https://ktor.io/servers/features/content-negotiation/serialization-converter.html

Upon using the example I get a deprecation warning for the „serialization(...“ Block because it has been replaced by json. Second the example does not work. In contrast, the gson example works perfectly fine. Either fix or remove the section, so other people don’t waste their time trying to get kotlinx.serializtion to work.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/224

The documentation for Micrometer metrics hint towards a Prometheus Registry being available for use here - https://github.com/ktorio/ktorio.github.io/blob/3e1f9c9c52ebbed7f429a05aa1c2b41ed69c6e93/servers/features/metrics-micrometer.md#L97

However, it appears that https://github.com/ktorio/ktor/pull/1105 has stalled out. Can you provide advise on if this feature will still be implemented, if the documentation needs to be updated, or something different. I would also like to know how I can help push this imitative along.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1842

Subsystem Server - metrics

Is your feature request related to a problem? Please describe. https://ktor.io/servers/features/metrics-micrometer.html

• I'm new to ktor and trying to have prometheus metrics, I do it in java app servers with simpleclient_servlet very easy
• I read install MicrometerMetrics, PrometheusMeterRegistry -> in doc missing dependency
• There is no doc where the metrics are exposed(URI)
• There is no doc if metrics need auth and which type of auth
• There is no doc should I need a route which expose metrics

Describe the solution you'd like

• More clear docs or working sample

Motivation to include to ktor Monitoring of app is very important and prometheus is very popular

In this section of the doc page https://ktor.io/docs/clients-index.html#requests-responses - there are two hyperlinks; one for sending requests and one for handling responses.

The "how to receive responses" link seems correct in that it links to the response handling section of the client docs https://ktor.io/docs/response.html. However, the "how to make requests" appears to link to a server side request handling page https://ktor.io/docs/requests.html#cookies instead of https://ktor.io/docs/request.html#specifying-a-body-for-requests.

On page https://ktor.io/docs/serialization-converter.html

Block:

install(ContentNegotiation) {
    serialization()
}

has deprecated code.

https://ktor.io/docs/http-client-engines.html#apache

Apache is the most configurable HTTP client about right now. It supports HTTP/1.1 and HTTP/2.

Unfortunately Ktor uses httpclient 4.4 but HTTP/2 support requires httpclient 5.

https://archive.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt https://hc.apache.org/httpcomponents-client-4.4.x/index.html

Perhaps I am missing something, but Ktor 1.4.1 is already released.

The same time public documentation for Ktor 1.4.0 says, that it's in development.

As for me, it looks like as it's not yet ready to be used:

❗ You are viewing Ktor 1.4.0 documentation, which is still in development.

Currently the panel enhances its size as new features are being added. But I believe the height of the right panel should equal to the max height of the whole window.

• testImplementation "io.ktor:ktor-server-tests:$ktor_version" : braces are missing
• implementation("io.ktor:ktor-server-core:$ktor_version)") : odd closing brace

If yo select Oauth feature it will fail to generate project either with a plugin or with a website. The problem is that it requires Locations feature but it is not aligned with all-features.json

See: https://ktor.io/docs/double-receive.html#usage

The native targets are always include, ignoring the variable. Looks like a wrong merge :)

https://github.com/ktorio/ktor/settings.gradle#L54

if (native_targets_enabled) {
    include ':ktor-client:ktor-client-curl'
    include ':ktor-client:ktor-client-ios'
}
include ':ktor-client:ktor-client-curl'
include ':ktor-client:ktor-client-ios'

Original question from https://stackoverflow.com/questions/64319112/why-serializing-collections-of-different-element-types-is-not-supported-in-ktor

To reproduce use the following code example:

@Serializable
sealed class Message {
    abstract val content: String
}

@Serializable
data class BroadcastMessage(override val content: String) : Message()

@Serializable
data class DirectMessage(override val content: String, val recipient: String) : Message()

val data: List<Message> = listOf(
    DirectMessage("Hey, Joe!", "Joe"),
    BroadcastMessage("Hey, all!")
)

fun main() {
    embeddedServer(Netty, port = 8080, host = "127.0.0.1") {
        install(ContentNegotiation) {
            json()
        }
        routing {
            get("/") {
                call.respond(data)
            }
        }
    }.start(wait = true)
}

Result:

java.lang.IllegalStateException: Serializing collections of different element types is not yet supported. Selected serializers: [DirectMessage, BroadcastMessage]
	at io.ktor.serialization.SerializerLookupKt.elementSerializer(SerializerLookup.kt:71)
	at io.ktor.serialization.SerializerLookupKt.serializerForSending(SerializerLookup.kt:41)
	at io.ktor.serialization.SerializationConverter.convertForSend(SerializationConverter.kt:114)
	at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:149)
	at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at ServerKt$main$1$2$1.invokeSuspend(server.kt:39)
	at ServerKt$main$1$2$1.invoke(server.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.routing.Routing.executeResult(Routing.kt:148)
	at io.ktor.routing.Routing.interceptor(Routing.kt:35)
	at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
	at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/129

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1885

Ktor Version and Engine Used (client or server and name) 1.3.2, Netty server, routes

ktor_version=1.3.2, kotlin_version=1.3.70

implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version")
    implementation("io.ktor:ktor-server-netty:$ktor_version")
    implementation("ch.qos.logback:logback-classic:$logback_version")
    testImplementation("io.ktor:ktor-server-tests:$ktor_version")

Describe the bug I migrated one of my project form php to ktor and deployed it in production for about 5k users. After one day server stopped working and linux file opened limits exceed which was 20k and when i checked with command ss -ant | grep :8383 | wc -l connections were 20k+ but i don't have users more than 5k which were not all active. Then i switched my floating ip to old php server and no more request to this server next day when i checked again with command ss -ant | grep :8383 | wc -l connections were still 20k+ even no more new request were coming and all connections were in ESTAB state.

So after that i created new project which was empty with ktor, kotlin dsl i mentioned its dependencies above and run that on that server Then i run this command watch -n 1 'ss -ant | grep {MY_IP}' on server to filter connections for my ip first it was no connection from my ip on port 8383 then i send single request from postman on SERVER_IP:8388 which was simple get request and response was from call.respond("Testing") and on server terminal it show me connection on 8383 with my ip. In postman request was complete and i got result. Then i closed postman and keep checking when it will close my connection from server. I checked that after an hour still connection was not closed and when i requested again it was still using that same connection. So it was reusing connection and not closing even after some hours. I'm not sure but may be error with keep alive or http2. I'm not sending any header i request.

I didn't tried any other ktor or kotlin version and I'm not able to reproduce it local. I'm running jar file generated with shadowjar on linux server with systemd service.

To Reproduce

1. create new project with ktor version 1.3.2 and kotlin 1.3.70 with GradleKotlinDsl

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(Routing) {
        get("/") {
            call.respond("Testing")
        }
    }
}

2. create jar file with shadowjar

tasks {
    named<ShadowJar>("shadowJar") {
        archiveBaseName.set("shadow")
        mergeServiceFiles()
        manifest {
            attributes(mapOf("Main-Class" to mainClasses))
        }
        transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer::class.java) {
            setPath("META-INF/services")
            include("org.eclipse.jetty.http.HttpFieldPreEncoder")
        }
        archiveFileName.set("example.jar")
    }
}

3. upload on ubuntu server and create its systemd service and run with systemctl start example service file:

[Unit]
Description=example
After=syslog.target

[Service]
User=root
LimitNOFILE=20536
ExecStart=/var/www/example/example-script
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

example-script:

#!/bin/sh
cd /var/www/example
java -jar exmaple.jar

Expected behavior When user make request to http://SERVER_IP:APP_PORT it should close it after request complete but it keep opened it until we will stop server. Checking with ss -ant | grep {CLIENT_PUBLIC_IP}

Screenshots This is screenshot which i get after an hour when made last request in screenshot last connection with port 8383 that is from my machine and it is still not closed.

Hi there,

We've discovered that Ktor 1.4.2 appears to crash out when sending large responses, using the Jackson feature. I think it might be related to KTOR-1342, but it's also happening on sending. Downgrading to ktor-jackson:1.4.1 seems to fix the issue.

The error logged is:

Exception in thread "eventLoopGroupProxy-3-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@2b5e16e4, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@114e04cb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
        at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
        at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
        at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
        at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
        at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
        at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
        at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
        at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
        at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
        at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)

I've created a sample project for you to try - https://github.com/SkyeWelch/ktor-1342-error-sample . You can run it with ./gradlew run, and if it's the same as on my machine see the following responses (I'm using httpie to make requests):

→ http localhost:8080/small
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked

{
    "string": "small string"
}

→ http localhost:8080/large
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked


http: error: ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Please let me know if you need anything else.

Thanks, Skye

As Ktor already has some overloads for kotlin.time.Duration in ktor-server-core/features/KotlinTimeJvm.kt, this overload would be nice too.

val date = GMTDate() + 1.hours

/**
 * Adds the specified [duration]
 */
@ExperimentalTime
public operator fun GMTDate.plus(duration: Duration): GMTDate = GMTDate(timestamp + duration.toLongMilliseconds())

/**
 * Subtracts the specified [duration]
 */
@ExperimentalTime
public operator fun GMTDate.minus(duration: Duration): GMTDate = GMTDate(timestamp - duration.toLongMilliseconds())

                of("route", call.attributes[measureKey].route ?: call.request.path()),

This line here (https://github.com/ktorio/ktor/blob/master/ktor-features/ktor-metrics-micrometer/jvm/src/io/ktor/metrics/micrometer/MicrometerMetrics.kt#L125) should not assign arbitrary value to a tag. The result of such behaviour is that if the application is maliciously scanned for every kind of URL possible, the application itself correctly returns 404 on such scans, but the metrics library still reports latency on every single one of the scans with its own tag value. This is very hard, for example, for influx, which has default limit on values per tag (https://stackoverflow.com/questions/43770354/max-values-per-tag-limit-exceeded-influxdb). And overall, storage of such information is useless.

So there should be some sane limit on possible values of route tag, possible limited by routes available in application.

Motivation from the PR: https://github.com/ktorio/ktor/pull/2034

I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load. I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs https://expressjs.com/en/resources/middleware/cors.html "allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex: Content-Type,Authorization) or an array (ex: ['Content-Type', 'Authorization']). If not specified, defaults to reflecting the headers specified in the request’s Access-Control-Request-Headers header."

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1767

Subsystem Server

Is your feature request related to a problem? Please describe. We want to use client-side certificates to authenticate HTTPS requests from connected devices. In our code, we need to get the client certificate information to determine the DN in the certificate.

In a regular servlet, client certificate information can be accessed using request.getAttribute("javax.servlet.request.X509Certificate"), which returns the certificate chain.

Our strategy to get this to work in ktor was to run our application as a WAR in a Tomcat instance configured to do client authentication. Unfortunately, request attributes do not seem to be available. (Note that a Spring Boot application deployed under the same Tomcat can access these attributes, so we know Tomcat is configured correctly for TLS.)

Describe the solution you'd like We'd like some way to access servlet request attributes from ktor application code.

Motivation to include to ktor Client certificates are an important part of IoT deployments, where ktor would shine because typically a server needs to manage a lot of concurrent connections. Ktor itself doesn't support client certificate authentication - our solution would be a workaround if only we could access request attributes.

Subsystem CIO - JVM

Motivation This fix adds the option to use only a specific alias instead all alias for a given JVM Keystore. This change is addictive only, no breaking change.

Solution Added null-able default null alias which will be use if present. Otherwise, all aliases from a Keystore will be used as known behavior. https://github.com/ktorio/ktor/pull/2230

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/82

If exception happens in ktor and is not handled by an interceptor such as StatusPages, it should propagate up to host and be handled there without using any ktor facilities like response pipeline. The reason is that we don't know which part of the system were at fault and it could very well be response pipeline itself. So unhandled exception should be propagated to the host-specific facilities and responded with 500 status code natively there.

Note, that it would make 500 Internal Server Error status produced in this case not interceptable by StatusPage feature on a per code basis, but exception filter in the same feature can be installed on Throwable and do pretty much any custom handling there. At this moment exception would be considered handled and won't be part of this issue.

From PR #540

Nowadays, it is quite common to pre-compress static files ( a.k.a assets ) to reduce response time (no need for on the fly compression).

This pull request enables that, serving the pre-compressed version of the file, suppressing compression(if installed) and responding appropriate Content-Type and Content-Encoding. As brotli is not supported at the time of writing, it also covers serving brotli-compressed files.

https://github.com/ktorio/ktor/pull/1880

Pull request here: https://github.com/ktorio/ktor/pull/2034

I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load. I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs https://expressjs.com/en/resources/middleware/cors.html "allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex: ‘Content-Type,Authorization’) or an array (ex: ['Content-Type', 'Authorization']). If not specified, defaults to reflecting the headers specified in the request’s Access-Control-Request-Headers header."

My understanding is that for a CORS simple request to run without prefight, part of the necessary criteria is that it can only have one of three ContentType values: [application/x-www-form-urlencoded, multipart/form-data, text/plain].

Issue KTOR-358 correctly added a check to ensure this criteria is met. But the current implementation doesn't consider when character encoding is appended to ContentType. For example ContentType: text/plain;charset=UTF-8. If the ContentType is specified in this manner, as is common/default on latest Chrome/Firefox, then the CORS feature will reject it, even though it is a simple request.

As far as I can tell from w3c there is no check required against the charset, rather the specification only cares about the mimeType. However, on reading mdn it says that the "only allowed values" are the three listed above which could be interpreted as not even charset is allowed.

As appending charset is the default on Chrome/Firefox, I think ktor should ignore parameters when parsing ContentType otherwise it will lead to devs setting allowNonSimpleContentTypes = true which isn't great.

The current CORS.kt feature rejects these requests at line 102:

public suspend fun intercept(context: PipelineContext<Unit, ApplicationCall>) {
    // ...
    if (!allowNonSimpleContentTypes) {
        val contentType = call.request.header(HttpHeaders.ContentType)?.let { ContentType.parse(it) }
        if (contentType != null) {
            if (contentType !in Configuration.CorsSimpleContentTypes) { // <--- issue
                context.respondCorsFailed()
                    return
                }
            }
        }
    //...
}

Configuration.CorsSimpleContentTypes has no values inside parameters, but the parsed value for contentType will contain charset=UTF-8. And so to fix this, then I think the check should be:

if(contentType.withoutParameters() !in Configuration.CorsSimpleContentTypes)

Thanks!

Since HTTP/2 push will be deprecated so we need to deprecate the corresponding API

Method io.ktor.http.auth.parseAuthorizationHeader(headerValue: String): HttpAuthHeader? throws IllegalArgumentException instead of returning null in some cases. Looking at the source code, the problem occurs because validation that the method does is weaker than the validation HttpAuthHeader.Single and HttpAuthHeader.Parametrized do in their constructors. In effect, when header has non-token68 characters in schema, the method throws when instantiating HttpAuthHeader.

To reproduce the issue, call the method with value, which includes non-token68 characters before first whitespace:

parseAuthorizationHeader("B<arer MyToken")

To fix the issue, the pattern checked for authScheme should discard values, which don't have token68 characters.

To reproduce run the following code:

val client = HttpClient(MockEngine) {
    engine {
        addHandler {
            respond("Test")
        }
    }
}

val engine = client.engine as MockEngine
engine.requestHistory
        .map(HttpRequestData::url)
        .map(Url::toString)
        .count()

I expect no exception, but the java.lang.IllegalStateException is thrown with Ktor 1.4.0:

Exception in thread "main" java.lang.IllegalStateException: Common concurrent list doesn't support iterator.
	at io.ktor.util.collections.ConcurrentList.iterator(ConcurrentList.kt:101)
	at MainKt.main(main.kt:26)
	at MainKt.main(main.kt)

No exception is thrown with Ktor 1.3.2-1.4.0-rc.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1876

From the beginning to Ktor 1.3.2.

Ktor makes no difference between URLs ending with a trailing slash or not.

The following code will produce an error because two handlers are trying to respond to the incoming request.

@Test fun `routing trailing slash`() = withTestApplication {
      application.routing {
          get("foo") {
              println("server processing /foo")
              call.respondText("/foo")
          }
          get("foo/") {
              println("server procession /foo/")
              call.respondText("/foo/")
          }
      }

      on ("making /foo request") {
          val result = handleRequest {
              uri = "/foo"
              method = HttpMethod.Get
          }
          it("should be handled") {
              assertTrue(result.requestHandled)
          }
          it("should have /foo as a response") {
              assertEquals("/foo", result.response.content)
          }
      }
  }


//produces
server processing /foo
server procession /foo/
Response has already been sent
io.ktor.server.engine.BaseApplicationResponse$ResponseAlreadySentException: Response has already been sent

This error has an impact on SEO because Google respects the difference between /foo and /foo/ that may produce different results.

It is also a problem when you want to use a first-level path parameter like "/{user}/" coupled with static content serving. Browsers can send requests to get a favicon file. These requests don't have any trailing slash, but still, Ktor will route them to the "/{user}/" handler instead of serving the file.

Expected behavior A router "//" should not handle a "/favicon.png" request.

Because it is a modification of behavior, users should activate this correction by configuration.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1691

Ktor Version and Engine Used (client or server and name) 1.3.1, client

Describe the bug Content-Disposition additional parameters should be inside quotes in multi-part Request body.

To Reproduce Steps to reproduce the behavior:

1. Multi-part request

import io.ktor.client.request.forms.append

<...>
httpClient.post<String> {
      body = MultiPartFormDataContent(formData {
        append(
          "file",
          "file.txt",
          ContentType.parse("text/plain")
        ) {
          writeText("content")
        }
      })
    }
<...>

2. Observe part data in Request body: Content-Disposition: form-data; name=file; filename=file.txt

Expected behavior Header matching spec:

The first parameter in the HTTP context is always form-data. Additional parameters are case-insensitive and have arguments that use quoted-string syntax after the '=' sign. Multiple parameters are separated by a semi-colon (';').

Content-Disposition: form-data; name="file"; filename="file.txt"

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1826

Subsystem

URLBuilder

Is your feature request related to a problem? Please describe.

I want to create a final URL from several paths. Let's say my starting url is https://example.com/items/ and incoming paths are: /category/, /directory/, /product.

Doing:

val url = URLBuilder("https://example.com/items/").run {
        path("/category/", "/directory/", "/product")
        buildString()
      }

I will end up with: https://example.com//category///directory///product. Notice that /items/ from starting URL is gone and the unnecessary slash duplications.

To preserve the original path, one can use:

 path(encodedPath, "/category/", "/directory/", "/product")

but that does not fix the path separator problem (in case where you don't control them).

Describe the solution you'd like

Adding a simple function, appendToPath, which will handle path extending and separators.

Exemplary implementation:

fun URLBuilder.appendToPath(vararg components: String) {
  // removes all trailing and leading separators
  val paths = components
    .map { it.replace(Regex("""(^/|/+$)"""), "") }
    .joinToString("/", transform = { it.encodeURLQueryComponent() })

  // make sure that there's a slash separator at the end of current path
  if (!encodedPath.endsWith('/')) {
    encodedPath = "${encodedPath}/"
  }
  this.encodedPath += paths
}

Motivation to include to ktor

I think this might be a common case (building up URL), so in my opinion, this small helper would be a beneficial addition. I'm more than happy to open up PR.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/771

Ktor Version

1.0.0

Ktor Engine Used(client or server and name)

Server: Netty Client: Apache Using OAuth feature

JVM Version, Operating System and Relevant Context

JVM: 10 OS: Mac

Feedback

OAuth2 with Dropbox fails during the "request access token" step. In summary, the OAuth feature includes the state parameter in the request. Dropbox's API does not allow unexpected parameters, and fails the OAuth access token request.

From what I can tell, there is currently no way to prevent the state from being added to the access token request.

Details

Consider the following 3 client-server exchanges. Here the Client is the Ktor server, and the Server is the identity provider (Dropbox, Google, Github, etc).

1. [Client] -- /authorize -> [Server]
2. [Client] <- ...?code=... -- [Server]
3. [Client] -- /access_token -> [Server]

In a typical OAuth2 exchange, the Client sends a state=<nonce> parameter with the request in 1 and the Server responds with state=<nonce> in the redirect in 2. (This state sending is not mandatory).

The OAuth feature is configured to include the state in 3 when the Client requests the access token from the Server. Some APIs such as Github's want the state to be included in the request. Dropbox however, reject requests that include the state.

The following code sets the state on the access token request. https://github.com/ktorio/ktor/blob/85210082c729b12099ad25f509e99b23ccea23b8/ktor-features/ktor-auth/src/io/ktor/auth/OAuth2.kt#L149-L151 In short, if a state was returned from the Server in 2, it will be non-null at this point in the code.

Using a custom OAuth2StateProvider

OAuth2ServerSettings allows the settings of a custom OAuth2StateProvider with which we can generate a custom state. I mentioned previously that the state is not mandatory, so maybe we can not generate a state? This is not an option, as the state must not be null. An empty state results in a request that has state=, but state is still there.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/975

Ktor Version

1.1.3

Ktor Engine Used(client or server and name)

EngineMain I guess?

JVM Version, Operating System and Relevant Context

JDK 1.8. Windows 10

Feedback

Okay I have a new project generated using the Ktor IDEA plugin. I'm just trying to make autoreload work (which should by default?). I have only one Kotlin file in src/Application.kt. My application.conf looks like this:

ktor {
    deployment {
        port = 8080
        port = ${?PORT}
        autoreload = true
        watch = [ MyProjectFolder ]
    }
    application {
        modules = [ com.company.ApplicationKt.module ]
    }
}

The issue is, when I change Application.kt the project doesn't reload and the changes are not reflected in the browser. I see this in the run output:

2019-02-25 11:01:46.894 [main] DEBUG Application - Watching F:\Projects\MyProjectFolder\out\production\classes for changes.
2019-02-25 11:01:46.895 [main] DEBUG Application - Watching F:\Projects\MyProjectFolder\out\production\classes\com for changes.
...

I also tried com.company instead of MyProjectFolder in the watch list but that one is even worse. The run output shows autoreload is disabled.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/813

Ktor Version

1.0.1

Ktor Engine Used(client or server and name)

Netty / Firefox

JVM Version, Operating System and Relevant Context

1.8 Windows Server 2008R2

Feedback

I don't how this happen, I can't reproduce it.

stacktrace:

500 Internal Server Error: POST - /search/TT
java.lang.IllegalStateException: Reading is not available in state Reading
        at kotlinx.coroutines.io.internal.ReadWriteBufferState.startReading$kotlinx_coroutines_io_jvm(ReadWriteBufferSta
te.kt:19)
        at kotlinx.coroutines.io.ByteBufferChannel.setupStateForRead(ByteBufferChannel.kt:238)
        at kotlinx.coroutines.io.ByteBufferChannel.access$setupStateForRead(ByteBufferChannel.kt:21)
        at kotlinx.coroutines.io.ByteBufferChannel.lookAhead(ByteBufferChannel.kt:2868)
        at kotlinx.coroutines.io.DelimitedKt.readUntilDelimiter(Delimited.kt:24)
        at io.ktor.http.cio.MultipartKt.copyUntilBoundary(Multipart.kt:339)
        at io.ktor.http.cio.MultipartKt$copyUntilBoundary$1.invokeSuspend(Multipart.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(Dispatched.kt:111)
        at kotlinx.coroutines.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:
83)
        at kotlinx.coroutines.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2207)
        at kotlinx.coroutines.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:865)
        at kotlinx.coroutines.io.ByteBufferChannel.readAsMuchAsPossible(ByteBufferChannel.kt:426)
        at kotlinx.coroutines.io.ByteBufferChannel.readAvailable(ByteBufferChannel.kt:559)
        at kotlinx.coroutines.io.ByteBufferChannel.readAvailableSuspend(ByteBufferChannel.kt:593)
        at kotlinx.coroutines.io.ByteBufferChannel$readAvailableSuspend$2.invokeSuspend(ByteBufferChannel.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:236)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

• Add release date for each release
• Add the link to the changelog on the main page