Changelog 1.5 version

Make the message of ResponseException contain response's text from response.readText().

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1930

Subsystem Client, JVM

Describe the solution you'd like In Java 11 was introduced HTTP Client based on reactive-streams.

Motivation to include to ktor Since more projects are switching to modern Java versions it could be valuable to use built-in Java HTTP Client without need to use additional libraries. It supports HTTP/1.1 and HTTP/2 protocol and provides async APIs.

Currently, Ktor calls OkHttpClient#newWebSocket to create an OkHttp WebSocket: https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L53

I would like to be able to provide my own instance of OkHttp WebSocket.Factory, so that I may override behaviour. Details of my use case are described here: https://github.com/square/okhttp/issues/6242

How do you feel about this? Thanks.

Provide/publish internal option to skip TLS verification for curl client engine

The following code fails with a ConnectTimeoutException. Maybe the property connectRetryAttempts is mislabeled and actually means the total number of attempts, not the number of retry attempts.

suspend fun main(): Unit = withContext(Dispatchers.Default) {
    HttpClient(CIO) {
        engine {
            endpoint.connectRetryAttempts = 0
        }
    }.use { client ->
        client.get<String>("https://www.google.com")
    }
}

Exception in thread "main" io.ktor.network.sockets.ConnectTimeoutException: Connect timeout has been expired [url=https://www.google.com/, connect_timeout=unknown ms]
	at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException(HttpTimeout.kt:164)
	at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException$default(HttpTimeout.kt:163)
	at io.ktor.client.engine.cio.Endpoint.getTimeoutException(Endpoint.kt:208)
	at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:200)
	at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Test to reproduce below. Tested on two different machines (Windows and macOS), works fine with CIO. With 500 test iterations, happens at least once most times.

import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.apache.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.http.*
import io.ktor.http.content.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.cio.*
import io.ktor.server.engine.*
import io.ktor.utils.io.*
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.RepeatedTest
import java.util.concurrent.TimeUnit

class KtorApacheBugTest {

    private val port = 8000
    private val server = embeddedServer(
        factory = CIO,
        port = port,
    ) {
        routing {
            put("/") {
                call.receiveChannel().discard()
                call.respond(HttpStatusCode.OK)
            }
        }
    }
    private val client = HttpClient(Apache) {
        install(HttpTimeout) {
            socketTimeoutMillis = 10_000
            connectTimeoutMillis = 10_000
        }
    }

    @BeforeEach
    fun setUp() {
        server.start()
    }

    @AfterEach
    fun tearDown() {
        client.close()
        server.stop(0, 1, TimeUnit.SECONDS)
    }

    @RepeatedTest(500)
    fun `eventually hits a socket timeout`(): Unit = runBlocking {
        val chunk = (0 until 10_000).map { it.toByte() }.toByteArray()

        client.put<String>("http://localhost:${port}") {
            body = ChannelWriterContent(
                body = {
                    writeFully(chunk)
                },
                contentType = null,
            )
        }
    }
}

Another simple test (seems the issue is with ChannelWriterContent) below.

This works:

@RepeatedTest(500)
    fun `does not hit a socket timeout`(): Unit = runBlocking {
        client.put<String>("http://localhost:${port}") {
            body = LocalFileContent(File("gradle/wrapper/gradle-wrapper.jar"))
        }
    }

This does not:

@RepeatedTest(500)
    fun `eventually hits a socket timeout`(): Unit = runBlocking {
        val chunk = File("gradle/wrapper/gradle-wrapper.jar").readBytes()

        client.put<String>("http://localhost:${port}") {
            body = ChannelWriterContent(
                body = {
                    writeFully(chunk)
                },
                contentType = null,
            )
        }
    }

Sending the same via ByteArrayContent works fine as well.

When doing get to e.g. only inspect the etag header or some other response header, the status code isn't validated. It's only validated when accessing the body. However, we often don't inspect the body, but only the header. This inconsistency has caused bugs on our side because we missed bad responses.

Could you please always validate the response status code, no matter if the body is accessed or not?

version: 1.4.3

when use a http proxy, get the exception.

use no proxy, it's ok.

code

val httpClient = HttpClient(CIO) {
    engine {
        proxy = httpProxy
    }
}

val c : String = httpClient.get(url)

build.gradle.kts

// ...

dependencies {
    implementation("org.jetbrains.kotlinx", "kotlinx-serialization-json", "1.0.1")
    implementation("io.ktor", "ktor-client-core", "1.4.3")
    implementation("io.ktor", "ktor-client-cio", "1.4.3")
    implementation("io.ktor", "ktor-client-serialization-jvm", "1.4.3")
    implementation("org.jetbrains.kotlinx", "kotlinx-coroutines-core", "1.4.2")

    // test
}

exception:

Channel was closed
kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
	at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
	at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
	at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
	at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
	at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
	at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
	(Coroutine boundary)
	at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)
	at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:76)
	at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
	(Coroutine creation stacktrace)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30)
	at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
	at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
	at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:128)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:92)
	at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:123)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at LoginTest$testLogin$1.invokeSuspend(LoginTest.kt:18)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at mytest(...)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:205)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:201)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:129)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
Caused by: kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
	at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
	at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
	at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
	at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
	at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
	at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Problem

Sometimes, we introduce new API but, due to our compatibility guarantees, we can't make it public and forced to introduce it as internal, sometimes with magic to invoke such internals from other modules. Usually, it is intended to make them public at some point in the future, however, because of long minor/major release cycles, it's easy to forget to promote such APIs.

Proposed solution

The solution is to provide an annotation with the following signature

/**
 * API marked with this annotation is intended to become public in the future [version].
 * Usually it means that the API can't be public at the moment of development due to
 * compatibility guarantees restrictions.
 *
 * Marking a public declaration with this annotation makes no sense
 * except for the case when it is also marked with [InternalAPI].

 * Please note that the specified [version] and the fact of making something a candidate is not a guarantee, 
 * so the target version could be changed without any notice or even the promotion could be cancelled at all.
 * 
 * @property version in which the API is planned to be promoted
 */
@InternalAPI // so users shouldn't use this annotation, it's for ktor only
@Retention(AnnotationRetention.SOURCE)
@Target(
    AnnotationTarget.CLASS,
    AnnotationTarget.ANNOTATION_CLASS,
    AnnotationTarget.CONSTRUCTOR,
    AnnotationTarget.FUNCTION,
    AnnotationTarget.PROPERTY,
    AnnotationTarget.PROPERTY_SETTER,
    AnnotationTarget.TYPEALIAS
)
public annotation class PublicAPICandidate(val version: String)

Using this annotation, one could mark some internal API, specifying the desired future version. It is recommended to document such API in spite of that it is internal yet.

When a minor/major version starts (or in the middle of development period), one should search for all annotation usages and collect/notify developers who wanted to make something public. At the end of the version development, we should ensure that there are no annotations with this version remaining.

Every promoted API should be properly documented and have adequate tests.

Example

In 1.4.1 we introduce a property

// In 1.4.1
public class SomeConfig {
    /** Initial documentation */
    @PublicAPICandidate("1.5.0") // the next minor version
    internal var usefulThing: String = ""
}

So no breaking changes, just the internal property.

When we develop 1.5.0 we search for all annotation usages and promote it removing annotation.

// In 1.5.0
public class SomeConfig {
    /**
     * Documentation
     */
    public var usefulProperty: String = ""
}

The other good effect is that by doing this we declare our intention to make something public and declare which version is it. So ktor users may see it, and this avoids unnecessary quesions and improves clarity.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/272

I tried the kotlinx serialization example from the docs: https://ktor.io/servers/features/content-negotiation/serialization-converter.html

Upon using the example I get a deprecation warning for the „serialization(...“ Block because it has been replaced by json. Second the example does not work. In contrast, the gson example works perfectly fine. Either fix or remove the section, so other people don’t waste their time trying to get kotlinx.serializtion to work.

This issue was imported from GitHub issue: https://github.com/ktorio/ktorio.github.io/issues/224

The documentation for Micrometer metrics hint towards a Prometheus Registry being available for use here - https://github.com/ktorio/ktorio.github.io/blob/3e1f9c9c52ebbed7f429a05aa1c2b41ed69c6e93/servers/features/metrics-micrometer.md#L97

However, it appears that https://github.com/ktorio/ktor/pull/1105 has stalled out. Can you provide advise on if this feature will still be implemented, if the documentation needs to be updated, or something different. I would also like to know how I can help push this imitative along.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1842

Subsystem Server - metrics

Is your feature request related to a problem? Please describe. https://ktor.io/servers/features/metrics-micrometer.html

• I'm new to ktor and trying to have prometheus metrics, I do it in java app servers with simpleclient_servlet very easy
• I read install MicrometerMetrics, PrometheusMeterRegistry -> in doc missing dependency
• There is no doc where the metrics are exposed(URI)
• There is no doc if metrics need auth and which type of auth
• There is no doc should I need a route which expose metrics

Describe the solution you'd like

• More clear docs or working sample

Motivation to include to ktor Monitoring of app is very important and prometheus is very popular

https://ktor.io/docs/http-client-engines.html#apache

Apache is the most configurable HTTP client about right now. It supports HTTP/1.1 and HTTP/2.

Unfortunately Ktor uses httpclient 4.4 but HTTP/2 support requires httpclient 5.

https://archive.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt https://hc.apache.org/httpcomponents-client-4.4.x/index.html

In this section of the doc page https://ktor.io/docs/clients-index.html#requests-responses - there are two hyperlinks; one for sending requests and one for handling responses.

The "how to receive responses" link seems correct in that it links to the response handling section of the client docs https://ktor.io/docs/response.html. However, the "how to make requests" appears to link to a server side request handling page https://ktor.io/docs/requests.html#cookies instead of https://ktor.io/docs/request.html#specifying-a-body-for-requests.

Perhaps I am missing something, but Ktor 1.4.1 is already released.

The same time public documentation for Ktor 1.4.0 says, that it's in development.

As for me, it looks like as it's not yet ready to be used:

❗ You are viewing Ktor 1.4.0 documentation, which is still in development.

On page https://ktor.io/docs/serialization-converter.html

Block:

install(ContentNegotiation) {
    serialization()
}

has deprecated code.

Currently the panel enhances its size as new features are being added. But I believe the height of the right panel should equal to the max height of the whole window.

• testImplementation "io.ktor:ktor-server-tests:$ktor_version" : braces are missing
• implementation("io.ktor:ktor-server-core:$ktor_version)") : odd closing brace

If yo select Oauth feature it will fail to generate project either with a plugin or with a website. The problem is that it requires Locations feature but it is not aligned with all-features.json

See: https://ktor.io/docs/double-receive.html#usage

The native targets are always include, ignoring the variable. Looks like a wrong merge :)

https://github.com/ktorio/ktor/settings.gradle#L54

if (native_targets_enabled) {
    include ':ktor-client:ktor-client-curl'
    include ':ktor-client:ktor-client-ios'
}
include ':ktor-client:ktor-client-curl'
include ':ktor-client:ktor-client-ios'

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1885

Ktor Version and Engine Used (client or server and name) 1.3.2, Netty server, routes

ktor_version=1.3.2, kotlin_version=1.3.70

implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version")
    implementation("io.ktor:ktor-server-netty:$ktor_version")
    implementation("ch.qos.logback:logback-classic:$logback_version")
    testImplementation("io.ktor:ktor-server-tests:$ktor_version")

Describe the bug I migrated one of my project form php to ktor and deployed it in production for about 5k users. After one day server stopped working and linux file opened limits exceed which was 20k and when i checked with command ss -ant | grep :8383 | wc -l connections were 20k+ but i don't have users more than 5k which were not all active. Then i switched my floating ip to old php server and no more request to this server next day when i checked again with command ss -ant | grep :8383 | wc -l connections were still 20k+ even no more new request were coming and all connections were in ESTAB state.

So after that i created new project which was empty with ktor, kotlin dsl i mentioned its dependencies above and run that on that server Then i run this command watch -n 1 'ss -ant | grep {MY_IP}' on server to filter connections for my ip first it was no connection from my ip on port 8383 then i send single request from postman on SERVER_IP:8388 which was simple get request and response was from call.respond("Testing") and on server terminal it show me connection on 8383 with my ip. In postman request was complete and i got result. Then i closed postman and keep checking when it will close my connection from server. I checked that after an hour still connection was not closed and when i requested again it was still using that same connection. So it was reusing connection and not closing even after some hours. I'm not sure but may be error with keep alive or http2. I'm not sending any header i request.

I didn't tried any other ktor or kotlin version and I'm not able to reproduce it local. I'm running jar file generated with shadowjar on linux server with systemd service.

To Reproduce

1. create new project with ktor version 1.3.2 and kotlin 1.3.70 with GradleKotlinDsl

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(Routing) {
        get("/") {
            call.respond("Testing")
        }
    }
}

2. create jar file with shadowjar

tasks {
    named<ShadowJar>("shadowJar") {
        archiveBaseName.set("shadow")
        mergeServiceFiles()
        manifest {
            attributes(mapOf("Main-Class" to mainClasses))
        }
        transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer::class.java) {
            setPath("META-INF/services")
            include("org.eclipse.jetty.http.HttpFieldPreEncoder")
        }
        archiveFileName.set("example.jar")
    }
}

3. upload on ubuntu server and create its systemd service and run with systemctl start example service file:

[Unit]
Description=example
After=syslog.target

[Service]
User=root
LimitNOFILE=20536
ExecStart=/var/www/example/example-script
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

example-script:

#!/bin/sh
cd /var/www/example
java -jar exmaple.jar

Expected behavior When user make request to http://SERVER_IP:APP_PORT it should close it after request complete but it keep opened it until we will stop server. Checking with ss -ant | grep {CLIENT_PUBLIC_IP}

Screenshots This is screenshot which i get after an hour when made last request in screenshot last connection with port 8383 that is from my machine and it is still not closed.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1767

Subsystem Server

Is your feature request related to a problem? Please describe. We want to use client-side certificates to authenticate HTTPS requests from connected devices. In our code, we need to get the client certificate information to determine the DN in the certificate.

In a regular servlet, client certificate information can be accessed using request.getAttribute("javax.servlet.request.X509Certificate"), which returns the certificate chain.

Our strategy to get this to work in ktor was to run our application as a WAR in a Tomcat instance configured to do client authentication. Unfortunately, request attributes do not seem to be available. (Note that a Spring Boot application deployed under the same Tomcat can access these attributes, so we know Tomcat is configured correctly for TLS.)

Describe the solution you'd like We'd like some way to access servlet request attributes from ktor application code.

Motivation to include to ktor Client certificates are an important part of IoT deployments, where ktor would shine because typically a server needs to manage a lot of concurrent connections. Ktor itself doesn't support client certificate authentication - our solution would be a workaround if only we could access request attributes.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/129

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/82

If exception happens in ktor and is not handled by an interceptor such as StatusPages, it should propagate up to host and be handled there without using any ktor facilities like response pipeline. The reason is that we don't know which part of the system were at fault and it could very well be response pipeline itself. So unhandled exception should be propagated to the host-specific facilities and responded with 500 status code natively there.

Note, that it would make 500 Internal Server Error status produced in this case not interceptable by StatusPage feature on a per code basis, but exception filter in the same feature can be installed on Throwable and do pretty much any custom handling there. At this moment exception would be considered handled and won't be part of this issue.

Subsystem CIO - JVM

Motivation This fix adds the option to use only a specific alias instead all alias for a given JVM Keystore. This change is addictive only, no breaking change.

Solution Added null-able default null alias which will be use if present. Otherwise, all aliases from a Keystore will be used as known behavior. https://github.com/ktorio/ktor/pull/2230

Pull request here: https://github.com/ktorio/ktor/pull/2034

I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load. I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs https://expressjs.com/en/resources/middleware/cors.html "allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex: ‘Content-Type,Authorization’) or an array (ex: ['Content-Type', 'Authorization']). If not specified, defaults to reflecting the headers specified in the request’s Access-Control-Request-Headers header."

Original question from https://stackoverflow.com/questions/64319112/why-serializing-collections-of-different-element-types-is-not-supported-in-ktor

To reproduce use the following code example:

@Serializable
sealed class Message {
    abstract val content: String
}

@Serializable
data class BroadcastMessage(override val content: String) : Message()

@Serializable
data class DirectMessage(override val content: String, val recipient: String) : Message()

val data: List<Message> = listOf(
    DirectMessage("Hey, Joe!", "Joe"),
    BroadcastMessage("Hey, all!")
)

fun main() {
    embeddedServer(Netty, port = 8080, host = "127.0.0.1") {
        install(ContentNegotiation) {
            json()
        }
        routing {
            get("/") {
                call.respond(data)
            }
        }
    }.start(wait = true)
}

Result:

java.lang.IllegalStateException: Serializing collections of different element types is not yet supported. Selected serializers: [DirectMessage, BroadcastMessage]
	at io.ktor.serialization.SerializerLookupKt.elementSerializer(SerializerLookup.kt:71)
	at io.ktor.serialization.SerializerLookupKt.serializerForSending(SerializerLookup.kt:41)
	at io.ktor.serialization.SerializationConverter.convertForSend(SerializationConverter.kt:114)
	at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:149)
	at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at ServerKt$main$1$2$1.invokeSuspend(server.kt:39)
	at ServerKt$main$1$2$1.invoke(server.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.routing.Routing.executeResult(Routing.kt:148)
	at io.ktor.routing.Routing.interceptor(Routing.kt:35)
	at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
	at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

https://github.com/ktorio/ktor/pull/1880

Motivation from the PR: https://github.com/ktorio/ktor/pull/2034

I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load. I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs https://expressjs.com/en/resources/middleware/cors.html "allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex: Content-Type,Authorization) or an array (ex: ['Content-Type', 'Authorization']). If not specified, defaults to reflecting the headers specified in the request’s Access-Control-Request-Headers header."

Since HTTP/2 push will be deprecated so we need to deprecate the corresponding API

Hi there,

We've discovered that Ktor 1.4.2 appears to crash out when sending large responses, using the Jackson feature. I think it might be related to KTOR-1342, but it's also happening on sending. Downgrading to ktor-jackson:1.4.1 seems to fix the issue.

The error logged is:

Exception in thread "eventLoopGroupProxy-3-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@2b5e16e4, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@114e04cb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
        at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
        at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
        at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
        at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
        at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
        at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
        at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
        at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
        at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
        at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)

I've created a sample project for you to try - https://github.com/SkyeWelch/ktor-1342-error-sample . You can run it with ./gradlew run, and if it's the same as on my machine see the following responses (I'm using httpie to make requests):

→ http localhost:8080/small
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked

{
    "string": "small string"
}

→ http localhost:8080/large
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked


http: error: ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Please let me know if you need anything else.

Thanks, Skye

My understanding is that for a CORS simple request to run without prefight, part of the necessary criteria is that it can only have one of three ContentType values: [application/x-www-form-urlencoded, multipart/form-data, text/plain].

Issue KTOR-358 correctly added a check to ensure this criteria is met. But the current implementation doesn't consider when character encoding is appended to ContentType. For example ContentType: text/plain;charset=UTF-8. If the ContentType is specified in this manner, as is common/default on latest Chrome/Firefox, then the CORS feature will reject it, even though it is a simple request.

As far as I can tell from w3c there is no check required against the charset, rather the specification only cares about the mimeType. However, on reading mdn it says that the "only allowed values" are the three listed above which could be interpreted as not even charset is allowed.

As appending charset is the default on Chrome/Firefox, I think ktor should ignore parameters when parsing ContentType otherwise it will lead to devs setting allowNonSimpleContentTypes = true which isn't great.

The current CORS.kt feature rejects these requests at line 102:

public suspend fun intercept(context: PipelineContext<Unit, ApplicationCall>) {
    // ...
    if (!allowNonSimpleContentTypes) {
        val contentType = call.request.header(HttpHeaders.ContentType)?.let { ContentType.parse(it) }
        if (contentType != null) {
            if (contentType !in Configuration.CorsSimpleContentTypes) { // <--- issue
                context.respondCorsFailed()
                    return
                }
            }
        }
    //...
}

Configuration.CorsSimpleContentTypes has no values inside parameters, but the parsed value for contentType will contain charset=UTF-8. And so to fix this, then I think the check should be:

if(contentType.withoutParameters() !in Configuration.CorsSimpleContentTypes)

Thanks!

Method io.ktor.http.auth.parseAuthorizationHeader(headerValue: String): HttpAuthHeader? throws IllegalArgumentException instead of returning null in some cases. Looking at the source code, the problem occurs because validation that the method does is weaker than the validation HttpAuthHeader.Single and HttpAuthHeader.Parametrized do in their constructors. In effect, when header has non-token68 characters in schema, the method throws when instantiating HttpAuthHeader.

To reproduce the issue, call the method with value, which includes non-token68 characters before first whitespace:

parseAuthorizationHeader("B<arer MyToken")

To fix the issue, the pattern checked for authScheme should discard values, which don't have token68 characters.

From PR #540

Nowadays, it is quite common to pre-compress static files ( a.k.a assets ) to reduce response time (no need for on the fly compression).

This pull request enables that, serving the pre-compressed version of the file, suppressing compression(if installed) and responding appropriate Content-Type and Content-Encoding. As brotli is not supported at the time of writing, it also covers serving brotli-compressed files.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1876

From the beginning to Ktor 1.3.2.

Ktor makes no difference between URLs ending with a trailing slash or not.

The following code will produce an error because two handlers are trying to respond to the incoming request.

@Test fun `routing trailing slash`() = withTestApplication {
      application.routing {
          get("foo") {
              println("server processing /foo")
              call.respondText("/foo")
          }
          get("foo/") {
              println("server procession /foo/")
              call.respondText("/foo/")
          }
      }

      on ("making /foo request") {
          val result = handleRequest {
              uri = "/foo"
              method = HttpMethod.Get
          }
          it("should be handled") {
              assertTrue(result.requestHandled)
          }
          it("should have /foo as a response") {
              assertEquals("/foo", result.response.content)
          }
      }
  }


//produces
server processing /foo
server procession /foo/
Response has already been sent
io.ktor.server.engine.BaseApplicationResponse$ResponseAlreadySentException: Response has already been sent

This error has an impact on SEO because Google respects the difference between /foo and /foo/ that may produce different results.

It is also a problem when you want to use a first-level path parameter like "/{user}/" coupled with static content serving. Browsers can send requests to get a favicon file. These requests don't have any trailing slash, but still, Ktor will route them to the "/{user}/" handler instead of serving the file.

Expected behavior A router "//" should not handle a "/favicon.png" request.

Because it is a modification of behavior, users should activate this correction by configuration.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1826

Subsystem

URLBuilder

Is your feature request related to a problem? Please describe.

I want to create a final URL from several paths. Let's say my starting url is https://example.com/items/ and incoming paths are: /category/, /directory/, /product.

Doing:

val url = URLBuilder("https://example.com/items/").run {
        path("/category/", "/directory/", "/product")
        buildString()
      }

I will end up with: https://example.com//category///directory///product. Notice that /items/ from starting URL is gone and the unnecessary slash duplications.

To preserve the original path, one can use:

 path(encodedPath, "/category/", "/directory/", "/product")

but that does not fix the path separator problem (in case where you don't control them).

Describe the solution you'd like

Adding a simple function, appendToPath, which will handle path extending and separators.

Exemplary implementation:

fun URLBuilder.appendToPath(vararg components: String) {
  // removes all trailing and leading separators
  val paths = components
    .map { it.replace(Regex("""(^/|/+$)"""), "") }
    .joinToString("/", transform = { it.encodeURLQueryComponent() })

  // make sure that there's a slash separator at the end of current path
  if (!encodedPath.endsWith('/')) {
    encodedPath = "${encodedPath}/"
  }
  this.encodedPath += paths
}

Motivation to include to ktor

I think this might be a common case (building up URL), so in my opinion, this small helper would be a beneficial addition. I'm more than happy to open up PR.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1691

Ktor Version and Engine Used (client or server and name) 1.3.1, client

Describe the bug Content-Disposition additional parameters should be inside quotes in multi-part Request body.

To Reproduce Steps to reproduce the behavior:

1. Multi-part request

import io.ktor.client.request.forms.append

<...>
httpClient.post<String> {
      body = MultiPartFormDataContent(formData {
        append(
          "file",
          "file.txt",
          ContentType.parse("text/plain")
        ) {
          writeText("content")
        }
      })
    }
<...>

2. Observe part data in Request body: Content-Disposition: form-data; name=file; filename=file.txt

Expected behavior Header matching spec:

The first parameter in the HTTP context is always form-data. Additional parameters are case-insensitive and have arguments that use quoted-string syntax after the '=' sign. Multiple parameters are separated by a semi-colon (';').

Content-Disposition: form-data; name="file"; filename="file.txt"

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/975

Ktor Version

1.1.3

Ktor Engine Used(client or server and name)

EngineMain I guess?

JVM Version, Operating System and Relevant Context

JDK 1.8. Windows 10

Feedback

Okay I have a new project generated using the Ktor IDEA plugin. I'm just trying to make autoreload work (which should by default?). I have only one Kotlin file in src/Application.kt. My application.conf looks like this:

ktor {
    deployment {
        port = 8080
        port = ${?PORT}
        autoreload = true
        watch = [ MyProjectFolder ]
    }
    application {
        modules = [ com.company.ApplicationKt.module ]
    }
}

The issue is, when I change Application.kt the project doesn't reload and the changes are not reflected in the browser. I see this in the run output:

2019-02-25 11:01:46.894 [main] DEBUG Application - Watching F:\Projects\MyProjectFolder\out\production\classes for changes.
2019-02-25 11:01:46.895 [main] DEBUG Application - Watching F:\Projects\MyProjectFolder\out\production\classes\com for changes.
...

I also tried com.company instead of MyProjectFolder in the watch list but that one is even worse. The run output shows autoreload is disabled.

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/771

Ktor Version

1.0.0

Ktor Engine Used(client or server and name)

Server: Netty Client: Apache Using OAuth feature

JVM Version, Operating System and Relevant Context

JVM: 10 OS: Mac

Feedback

OAuth2 with Dropbox fails during the "request access token" step. In summary, the OAuth feature includes the state parameter in the request. Dropbox's API does not allow unexpected parameters, and fails the OAuth access token request.

From what I can tell, there is currently no way to prevent the state from being added to the access token request.

Details

Consider the following 3 client-server exchanges. Here the Client is the Ktor server, and the Server is the identity provider (Dropbox, Google, Github, etc).

1. [Client] -- /authorize -> [Server]
2. [Client] <- ...?code=... -- [Server]
3. [Client] -- /access_token -> [Server]

In a typical OAuth2 exchange, the Client sends a state=<nonce> parameter with the request in 1 and the Server responds with state=<nonce> in the redirect in 2. (This state sending is not mandatory).

The OAuth feature is configured to include the state in 3 when the Client requests the access token from the Server. Some APIs such as Github's want the state to be included in the request. Dropbox however, reject requests that include the state.

The following code sets the state on the access token request. https://github.com/ktorio/ktor/blob/85210082c729b12099ad25f509e99b23ccea23b8/ktor-features/ktor-auth/src/io/ktor/auth/OAuth2.kt#L149-L151 In short, if a state was returned from the Server in 2, it will be non-null at this point in the code.

Using a custom OAuth2StateProvider

OAuth2ServerSettings allows the settings of a custom OAuth2StateProvider with which we can generate a custom state. I mentioned previously that the state is not mandatory, so maybe we can not generate a state? This is not an option, as the state must not be null. An empty state results in a request that has state=, but state is still there.

• Add release date for each release
• Add the link to the changelog on the main page