Changelog 1.4 version
1.4.3
released 1st December 2020
Client
ClassCastException with HTTP client when using via an inline function with reified type parameter
ktor HTTP client results in a ClassCastException
("class java.util.LinkedHashMap
cannot be cast to class MyEnvironmentDetails
").
suspend fun main() {
InternalClient().use {
it.getEnvironment()
}
}
@KtorExperimentalAPI
class MyClient: Closeable {
private val client: HttpClient = HttpClient(CIO) {
install(JsonFeature) {
serializer = JacksonSerializer {
registerModule(JavaTimeModule())
}
}
}
suspend fun getEnvironment(): MyEnvironmentDetails {
return internalRequest("Core.GetEnvironment")
}
private suspend inline fun <reified T> internalRequest(method: String): T {
val rawResponse = client.post<RawApiResponse<T>> {
url("https", "my.site", 443, "/api")
accept(ContentType.Application.Json)
contentType(ContentType.Application.Json)
body = ApiRequest(method, Unit)
}
return rawResponse.outParams
}
override fun close() {
client.close()
}
}
data class ApiRequest<T>(
val action: String,
val inParams: T,
) {
val apiVersion: BigDecimal = 2.0.toBigDecimal()
}
@JsonIgnoreProperties(ignoreUnknown = true)
data class RawApiResponse<T>(
val outParams: T,
val sessionTrackingId: String?,
)
@JsonIgnoreProperties(ignoreUnknown = true)
data class MyEnvironmentDetails(
val lang: String,
val title: String,
)
If I manually inline internalRequest()
, e.g. so that getEnvironment()
becomes:
suspend fun getEnvironment(): MyEnvironmentDetails {
val rawResponse = client.post<RawApiResponse<MyEnvironmentDetails>> {
url("https", "my.site", 443, "/api")
accept(ContentType.Application.Json)
contentType(ContentType.Application.Json)
body = ApiRequest("Core.GetEnvironment", Unit)
}
return rawResponse.outParams
}
It works as expected (i.e. no ClassCastException
). Tracking this down through the debugger I can see that the io.ktor.client.call.TypeInfo<T>
in my initial example has a reifiedType
that represents a RawApiResponse<T>
and the working example has a reifiedType
that represents a RawApiResponse< MyEnvironmentDetails>
.
Can I logging single line?
Hi There.
I'am try to using ktor-client-logging-jvm
.
This logging module display multiple line like this and so many line.
HttpClient: REQUEST: http://localhost:8081/testPost
HttpClient: METHOD: HttpMethod(value=POST)
HttpClient: COMMON HEADERS
HttpClient: -> Accept: application/json
HttpClient: -> Accept-Charset: UTF-8
HttpClient: -> User-Agent: Apache-HttpAsyncClient/4.5.3(npay-point)
HttpClient: CONTENT HEADERS
HttpClient: -> Content-Length: 12
HttpClient: -> Content-Type: text/plain; charset=UTF-8
HttpClient: BODY Content-Type: text/plain; charset=UTF-8
HttpClient: BODY START
HttpClient: asdfasdfasdf
HttpClient: BODY END
HttpClient: RESPONSE: 502
HttpClient: METHOD: HttpMethod(value=POST)
HttpClient: FROM: http://localhost:8081/testPost
HttpClient: COMMON HEADERS
HttpClient: -> Connection: keep-alive, keep-alive
HttpClient: -> Content-Length: 11
HttpClient: -> Content-Type: text/plain;charset=UTF-8
HttpClient: -> Date: Fri, 04 Dec 2020 08:14:04 GMT
HttpClient: -> Keep-Alive: timeout=60
HttpClient: BODY Content-Type: text/plain; charset=UTF-8
HttpClient: BODY START
HttpClient: Post Result
HttpClient: BODY END
Can I logging this to single line like this?
Request : Url[http....], Headers, body, params
Response : Header, body
and if logging with elapsed time. it is very helpful.
Thank u.
iOS testing MockEngine issue
My goal is to mock the httpClient to test several error cases on Android and iOS (since there are different errors on each platform).
For Android everything work fine, but for iOS I am still facing the same error:
kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen ***.shared.MockEngine
I have tested a lot of iosTest dependencies but didn't succeed to find the good ktor-client-mock.
the last I tested was this one:
val iosTest by getting {
dependencies {
implementation("io.ktor:ktor-client-mock-iosx64:$ktorVersion")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core-iosx64:$coroutinesVersion")
}
}
I am using kotlin 1.4.0
and Ktor 1.4.3
I am running iosTest using :
val iosTest: Task by tasks.creating {
val device = project.findProperty("iosDevice")?.toString() ?: "iPhone X"
val testExecutable =
kotlin.targets.getByName<KotlinNativeTarget>("iosX64").binaries.getTest("DEBUG")
dependsOn(testExecutable.linkTaskName)
group = JavaBasePlugin.VERIFICATION_GROUP
description = "Runs tests for target 'ios' on an iOS simulator"
doLast {
exec {
println(testExecutable.outputFile.absolutePath)
commandLine(
"xcrun",
"simctl",
"spawn",
"--standalone",
device,
testExecutable.outputFile.absolutePath
)
}
}
}
ClassCastException: kotlin.coroutines.intrinsics.CoroutineSingletons cannot be cast to ...
Hi there. what is a good way to approach debugging this?
I am using suspend fun as well as Flows and awaits, but I am not sure where to start with this.
on Slack, Roman suggested this might be ktor related.
Exception in thread "main" java.lang.ClassCastException: kotlin.coroutines.intrinsics.CoroutineSingletons cannot be cast to io.ktor.client.statement.HttpResponse
at com.github.wakingrufus.kalibrate.example.MainKt$main$2$4$1$1$1.invoke(Main.kt:68)
at com.github.wakingrufus.kalibrate.example.MainKt$main$2$4$1$1$1.invoke(Main.kt)
at com.github.wakingrufus.kalibrate.scenario.Step.invoke(Step.kt:15)
at com.github.wakingrufus.kalibrate.scenario.StepContainer$invoke$1.invokeSuspend(Simulation.kt:93)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:86)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:61)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at com.github.wakingrufus.kalibrate.scenario.StepContainer.invoke(Simulation.kt:91)
at com.github.wakingrufus.kalibrate.scenario.Simulation.runSetup(Simulation.kt:75)
at com.github.wakingrufus.kalibrate.scenario.Simulation$singletonWorkPattern$1$1$1.invokeSuspend(Simulation.kt:25)
at com.github.wakingrufus.kalibrate.scenario.Simulation$singletonWorkPattern$1$1$1.invoke(Simulation.kt)
at kotlinx.coroutines.flow.SafeFlow.collectSafely(Builders.kt:61)
at kotlinx.coroutines.flow.AbstractFlow.collect(Flow.kt:212)
at kotlinx.coroutines.flow.FlowKt__MergeKt$flattenConcat$$inlined$unsafeFlow$1.collect(SafeCollector.common.kt:114)
at kotlinx.coroutines.flow.internal.ChannelFlowMerge$collectTo$$inlined$collect$1$lambda$1.invokeSuspend(Merge.kt:69)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:86)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:61)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at com.github.wakingrufus.kalibrate.KalibrateDslBuilder.invoke(KalibrateDsl.kt:92)
at com.github.wakingrufus.kalibrate.KalibrateDslKt$kalibrate$1.invoke(KalibrateDsl.kt:125)
at com.github.wakingrufus.kalibrate.KalibrateDslKt$kalibrate$1.invoke(KalibrateDsl.kt)
at com.xenomachina.argparser.SystemExitExceptionKt.mainBody(SystemExitException.kt:74)
at com.xenomachina.argparser.SystemExitExceptionKt.mainBody$default(SystemExitException.kt:72)
at com.github.wakingrufus.kalibrate.KalibrateDslKt.kalibrate(KalibrateDsl.kt:124)
at com.github.wakingrufus.kalibrate.example.MainKt.main(Main.kt:16)
source:
https://github.com/wakingrufus/kalibrate
to reproduce:
cd kalibrate-example/
./example.sh --scenario deploy
'%3D' inside query of redirect target location will be replaced to '='
When the server returns 302 with the URL has '%3D' (inside query part), the Ktor replaces '%3D' with '=' so some server returns invalid content (in this case, 403).
Test code:
import io.ktor.client.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.runBlocking
fun main(): Unit = runBlocking {
val url = "https://dl.bintray.com/anatawa12/maven/com/anatawa12/forge/ForgeGradle/1.2-1.0.0/ForgeGradle-1.2-1.0.0-javadoc.jar"
// also in CIO
//val factory = io.ktor.client.engine.cio.CIO
val factory = io.ktor.client.engine.apache.Apache
val clientNoFollow = HttpClient(io.ktor.client.engine.cio.CIO) {
followRedirects = false
expectSuccess = false
}
val response = clientNoFollow.get<HttpResponse>(url)
println("status: " + response.status)
println("location: " + response.headers["Location"])
println("location has %3D: " + response.headers["Location"]!!.contains("%3D"))
check(response.headers["Location"]!!.contains("%3D"))
val clientWithFollow = HttpClient(io.ktor.client.engine.cio.CIO)
try {
val got = clientWithFollow.get<ByteArray>(url)
} catch (e: ResponseException) {
println("error: " + e)
println("error url: " + e.response.request.url)
println("error url has %3D: " + e.response.request.url.toString().contains("%3D"))
}
}
Log:
status: 302
location: https://d29vzk4ow07wi7.cloudfront.net/d133d0ab57f0e1a996fa1367eb3e5c7ebbd1860a5d505a69791d7746457acb0a?response-content-disposition=attachment%3Bfilename%3D%22ForgeGradle-1.2-1.0.0-javadoc.jar%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvZDEzM2QwYWI1N2YwZTFhOTk2ZmExMzY3ZWIzZTVjN2ViYmQxODYwYTVkNTA1YTY5NzkxZDc3NDY0NTdhY2IwYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMkZvcmdlR3JhZGxlLTEuMi0xLjAuMC1qYXZhZG9jLmphciUyMiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYxMjkzNzUyOX0sIklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiIwLjAuMC4wLzAifX19XX0_&Signature=ch~WD7z9cBluu3KWX2pIsmCd7pfVQ~WnM2QEg6wLuDPFi5pr2e4DnHApdAX9A92dxJW68WYllbKB1pNRkJ6RjtW~3eip545DnUcSBYGT1vfiaxNzp80QYsg~EUo~l4eHFYVoywXXPNgEPRXPIZ2vHGv7H7aeGJ5fbhzyunSmdstRnfUpfKLL5XmsDjpvVY4RYy1NSHHG70BU3y6UP1nqdu1hEdzBhRFKwZ~0uj8~2S4OF6Daaj71jkRWF49TQS~aNpvrkaAZojUilWEwppXZNcCekXu7Bt95oJDESsNQf0bu0gDw5eNtAfy2ZzN~a~lXVTV3ORy3kVanNwiVJzcf~g__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA
location has %3D: true
error: io.ktor.client.features.ClientRequestException: Client request(https://d29vzk4ow07wi7.cloudfront.net/d133d0ab57f0e1a996fa1367eb3e5c7ebbd1860a5d505a69791d7746457acb0a?response-content-disposition=attachment%3Bfilename=%22ForgeGradle-1.2-1.0.0-javadoc.jar%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvZDEzM2QwYWI1N2YwZTFhOTk2ZmExMzY3ZWIzZTVjN2ViYmQxODYwYTVkNTA1YTY5NzkxZDc3NDY0NTdhY2IwYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMkZvcmdlR3JhZGxlLTEuMi0xLjAuMC1qYXZhZG9jLmphciUyMiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYxMjkzNzUzMH0sIklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiIwLjAuMC4wLzAifX19XX0_&Signature=XwmQYLdJSkb~0fNr36ebj1vzn1KXmufRtaP9Mj9y9thRAtpAbyImJU-31VhtEhV56ijN2-tBtT5m0rNPSq-f6AEvngoEZURUUpE-vGVb1UFwiGnVnTuDKQJUvao3gZTj-fq9kRCLosJGYJiVbLG3GTXQKYoxL9fQh2ttwS3eFKAyFylcjyvsmSkEAekomfjB8OhUpgc~B6CE0Vg6aOm3ZBaKNAtfX4~cccnFrz4PB836WD68y8qX9jN5IQMCAiDy0881chyPIjGM7c30aycANw5c8StHSRy~gWo5WL7TGpq19X5q2pK-sT8Vq0I3AJJZ95XW9N3GWj0WR7IKqOQTAg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA) invalid: 403 Forbidden. Text: "<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access denied</Message></Error>"
error url: https://d29vzk4ow07wi7.cloudfront.net/d133d0ab57f0e1a996fa1367eb3e5c7ebbd1860a5d505a69791d7746457acb0a?response-content-disposition=attachment%3Bfilename=%22ForgeGradle-1.2-1.0.0-javadoc.jar%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvZDEzM2QwYWI1N2YwZTFhOTk2ZmExMzY3ZWIzZTVjN2ViYmQxODYwYTVkNTA1YTY5NzkxZDc3NDY0NTdhY2IwYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMkZvcmdlR3JhZGxlLTEuMi0xLjAuMC1qYXZhZG9jLmphciUyMiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTYxMjkzNzUzMH0sIklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiIwLjAuMC4wLzAifX19XX0_&Signature=XwmQYLdJSkb~0fNr36ebj1vzn1KXmufRtaP9Mj9y9thRAtpAbyImJU-31VhtEhV56ijN2-tBtT5m0rNPSq-f6AEvngoEZURUUpE-vGVb1UFwiGnVnTuDKQJUvao3gZTj-fq9kRCLosJGYJiVbLG3GTXQKYoxL9fQh2ttwS3eFKAyFylcjyvsmSkEAekomfjB8OhUpgc~B6CE0Vg6aOm3ZBaKNAtfX4~cccnFrz4PB836WD68y8qX9jN5IQMCAiDy0881chyPIjGM7c30aycANw5c8StHSRy~gWo5WL7TGpq19X5q2pK-sT8Vq0I3AJJZ95XW9N3GWj0WR7IKqOQTAg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA
error url has %3D: false
NPE in ApacheRequestProducer when "http://" is requested
The following exception is being thrown when I'm trying to send a request to "http://"
java.lang.NullPointerException
at io.ktor.client.engine.apache.ApacheRequestProducer.<init>(ApacheRequestProducer.kt:38)
at io.ktor.client.engine.apache.ApacheEngine.execute(ApacheEngine.kt:36)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Digest authentition: using the realm of the 401 response to an initial incomplete request for an immediate second complete request including realm.
There are cases where the client of a digest authentication has no knowledge of the realm value. For this scenario, a case in which the realm contained in the 401 response is immediately used should be added.
ResponseException is no longer serializable starting from 1.4.0 (breaking change)
Previously a ResponseException could be serialized. See: https://github.com/ktorio/ktor/issues/1256. The Transient annotation was removed in a recent commit for the HttpResponse
field. Now an exception is thrown when serializing a ResponseException.
Stack trace:
io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
java.io.NotSerializableException: io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
at ExceptionsTest.serialize(ExceptionsTest.kt:26)
at ExceptionsTest.testResponseExceptionSerializable(ExceptionsTest.kt:17)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Here is a test to reproduce the issue:
import io.ktor.client.call.*
import io.ktor.client.features.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.util.date.*
import io.ktor.utils.io.*
import java.io.*
import kotlin.coroutines.*
import kotlin.test.*
class ExceptionsTest {
@Test
fun testResponseExceptionSerializable() {
val exception = createResponseException()
val serialized = serialize(exception)
val deserialized = deserialize(serialized)
deserialized as ResponseException
}
private fun serialize(obj: Any): ByteArray {
val b = ByteArrayOutputStream()
val o = ObjectOutputStream(b)
o.writeObject(obj)
return b.toByteArray()
}
private fun deserialize(bytes: ByteArray): Any? {
val b = ByteArrayInputStream(bytes)
val o = ObjectInputStream(b)
return o.readObject()
}
}
private fun createResponseException(): ResponseException = ResponseException(object : HttpResponse() {
override val call: HttpClientCall
get() = TODO("Not yet implemented")
override val status: HttpStatusCode
get() = TODO("Not yet implemented")
override val version: HttpProtocolVersion
get() = TODO("Not yet implemented")
override val requestTime: GMTDate
get() = TODO("Not yet implemented")
override val responseTime: GMTDate
get() = TODO("Not yet implemented")
override val content: ByteReadChannel
get() = TODO("Not yet implemented")
override val headers: Headers
get() = TODO("Not yet implemented")
override val coroutineContext: CoroutineContext
get() = TODO("Not yet implemented")
override fun toString(): String = "FakeCall"
}, cachedResponseText = "Fake text")
ClosedReceiveChannelException when making request with CIO engine using a proxy to https
version: 1.4.3
when use a http proxy, get the exception.
use no proxy, it's ok.
code
val httpClient = HttpClient(CIO) {
engine {
proxy = httpProxy
}
}
val c : String = httpClient.get(url)
build.gradle.kts
// ...
dependencies {
implementation("org.jetbrains.kotlinx", "kotlinx-serialization-json", "1.0.1")
implementation("io.ktor", "ktor-client-core", "1.4.3")
implementation("io.ktor", "ktor-client-cio", "1.4.3")
implementation("io.ktor", "ktor-client-serialization-jvm", "1.4.3")
implementation("org.jetbrains.kotlinx", "kotlinx-coroutines-core", "1.4.2")
// test
}
exception:
Channel was closed
kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
(Coroutine boundary)
at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)
at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:76)
at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:81)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
(Coroutine creation stacktrace)
at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30)
at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:128)
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:92)
at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:123)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at LoginTest$testLogin$1.invokeSuspend(LoginTest.kt:18)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at mytest(...)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:205)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:201)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:129)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at mytest(...)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at mytest(...)
at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at myfunction(...)
at mytest(...)
Caused by: kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Client: NPE in FormDataContentKt -> Input.copyTo
I've received the following exception report for an Android app.
It was sporadic and I cannot reproduce the problem. The problem seems to lie within Ktor itself so I'll simply share what I can.
FATAL EXCEPTION: k.a0 Dispatcher = okhttp3.OkHttpClient Dispatcher
java.lang.NullPointerException
at h.b.a.g.j.b$a.l(Unknown Source:58) = io.ktor.client.request.forms.FormDataContentKt$copyTo$2.invokeSuspend(java.lang.Object)
at kotlin.c0.k.a.a.n(Unknown Source:9) = kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(java.lang.Object)
at kotlinx.coroutines.a1.run(Unknown Source:93)
at kotlinx.coroutines.d3.b.b0(Unknown Source:0)
at kotlinx.coroutines.d3.b$a.c(Unknown Source:14)
at kotlinx.coroutines.d3.b$a.m(Unknown Source:28)
at kotlinx.coroutines.d3.b$a.run(Unknown Source:0)
This is the relevant mapping:
io.ktor.client.request.forms.FormDataContentKt$copyTo$2 -> h.b.a.g.j.b$a:
io.ktor.utils.io.WriterSuspendSession p$ -> p
java.lang.Object L$0 -> q
io.ktor.utils.io.core.Input $this_copyTo -> s
int label -> r
java.lang.Object invoke(java.lang.Object,java.lang.Object) -> M
kotlin.coroutines.Continuation create(java.lang.Object,kotlin.coroutines.Continuation) -> d
java.lang.Object invokeSuspend(java.lang.Object) -> l
The relevant code from Ktor:
https://github.com/ktorio/ktor/blob/a6d68420ee0b272cd09d235057427a6601736487/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/FormDataContent.kt#L154
private suspend fun Input.copyTo(channel: ByteWriteChannel) {
if (this is ByteReadPacket) {
channel.writePacket(this)
return
}
channel.writeSuspendSession {
while (!this@copyTo.endOfInput) {
tryAwait(1)
val buffer = request(1)!! // <-- likely cause
val size = this@copyTo.readAvailable(buffer)
if (size < 0) continue
written(size)
}
}
}
CIO: client engine exceptions are both logged and thrown
In the CIO client engine exceptions like SocketTimeoutException
are logged to standard error by the coroutines' uncaught exception handler and thrown by client.get()
.
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.*
suspend fun main(): Unit = withContext(Dispatchers.Default) {
HttpClient(CIO) {
install(HttpTimeout) {
socketTimeoutMillis = 5
}
}.use { client ->
try {
client.get<HttpResponse>("http://www.google.com").readText()
} catch (e: Throwable) {
println("Caught: $e")
}
}
}
Output:
Exception in thread "kotlinx.coroutines.DefaultExecutor" java.net.SocketTimeoutException
at io.ktor.network.util.UtilsKt$withSocketTimeout$2.invokeSuspend(Utils.kt:23)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32)
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:188)
at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:122)
at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:111)
at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)
at kotlinx.coroutines.CancellableContinuationImpl.cancel(CancellableContinuationImpl.kt:177)
at kotlinx.coroutines.CancellableContinuationImpl.parentCancelled$kotlinx_coroutines_core(CancellableContinuationImpl.kt:184)
at kotlinx.coroutines.ChildContinuation.invoke(JobSupport.kt:1484)
at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:1511)
at kotlinx.coroutines.JobSupport.tryMakeCancelling(JobSupport.kt:792)
at kotlinx.coroutines.JobSupport.makeCancelling(JobSupport.kt:752)
at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:668)
at kotlinx.coroutines.JobSupport.cancelCoroutine(JobSupport.kt:655)
at kotlinx.coroutines.TimeoutCoroutine.run(Timeout.kt:128)
at kotlinx.coroutines.EventLoopImplBase$DelayedRunnableTask.run(EventLoop.common.kt:497)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:68)
at java.base/java.lang.Thread.run(Thread.java:832)
Caught: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=http://www.google.com/, socket_timeout=5] ms
HttpRedirect feature alters Location header value
Currently HttpRedirect
feature alters Location
header value by escaping query parameters, but it's significant in case of signed URLs verifications.
val httpClient = HttpClient(Apache) {
followRedirects = true
}
val response = httpClient.get<HttpResponse>("https://dl.bintray.com/jetbrains/markdown/org/jetbrains/markdown-jvm/0.2.0.pre-61/markdown-jvm-0.2.0.pre-61.jar")
assertTrue(response.status.isSuccess())
The original response contains Location
like that:
https://akamai.bintray.com/22/225b067044aa56f36590ef56d41e256cd1d0887b176bfdeec123ecccc6057790?__gda__=exp=1604350711~hmac=417cbd5a97b4c499e2cf7e9eae5dfb9ad95b42cb3ff76c5fb0fae70e2a42db9c&...
But Ktor executes URL encode for query parameters and uses the following value:
https://akamai.bintray.com/22/225b067044aa56f36590ef56d41e256cd1d0887b176bfdeec123ecccc6057790?__gda__=exp%3D1604350711~hmac%3D417cbd5a97b4c499e2cf7e9eae5dfb9ad95b42cb3ff76c5fb0fae70e2a42db9c&...
CIO Engine's HttpClient may fail when trying to send large size binary data.
Ktor Version and Engine Used (client or server and name)
Client CIO, 1.4.1
Describe the bug
HttpClient with CIO Engine sometimes sends an invalid multipart/form-data request when it includes large size binary data.
When it comes to invalid requests, it does not seem to contain a trailing multipart/form-data request boundary.
(There doesn't seem to be enough content in the request body.)
java.nio.BufferOverflowException
seems to occur in the following process.
https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/FormDataContent.kt#L123
This behavior is not reproduced in the case of HttpClient with Apache Engine.
To Reproduce
Sample code is below.
@KtorExperimentalAPI
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
routing {
post("/multipart") {
val multipartData = call.receiveMultipart()
multipartData.forEachPart {
if (it is PartData.FileItem) {
println(
withContext(Dispatchers.IO) {
it.streamProvider().readAllBytes().size
}
)
}
}
call.respondText("test")
}
get("/execute/cio") {
repeat(10) {
request(HttpClient(CIO))
}
call.respondText("success")
}
get("/execute/apache") {
repeat(10) {
request(HttpClient(Apache))
}
call.respondText("success")
}
}
}
suspend fun request(client: HttpClient) {
val body = formData {
val headers = headersOf(
HttpHeaders.ContentDisposition,
ContentDisposition.File
.withParameter(ContentDisposition.Parameters.Name, "file")
.withParameter(ContentDisposition.Parameters.FileName, "test.jpg")
.toString()
)
appendInput("file", headers) {
// dummy 10MB binary file
ByteArray(10_000_000).inputStream().asInput()
}
}
client.use {
it.submitFormWithBinaryData<Unit>(
url = "http://localhost:8080/multipart",
formData = body
)
}
}
(full code is in https://github.com/ma2Cta/ktor-multipart-request-test/blob/main/src/Application.kt )
-
Start the server.
-
Run
$ curl http://localhost:8080/execute/apache
a few times.
10 times 10000000
will be printed as expected.
- Run
$ curl http://localhost:8080/execute/cio
a few times.
You won't get 10000000
10 times, and you'll probably get a number less than 10000000
the first few times
Expected
The correct request is sent even in the case of CIO Engine.
Timeout feature: android engine throws Java's SocketTimeoutException instead of ConnectTimeoutException
While using Ktor client, with timeout feature, the following instruction can throw a java.net.SocketTimeoutException
instead of the expected ConnectTimeoutException
. The stack trace is in the attachments.
PS: The exceptions section in the Timeout feature documentation is outdated.
Client sometimes doesn't do response validation
When doing get<HttpResponse> to e.g. only inspect the etag header or some other response header, the status code isn't validated. It's only validated when accessing the body. However, we often don't inspect the body, but only the header. This inconsistency has caused bugs on our side because we missed bad responses.
Could you please always validate the response status code, no matter if the body is accessed or not?
Make `body` nullable for request builder
Currently in the HttpClient, when building a request, if you want to set the body to empty, you have to do body = EmptyContent
.
Why not just have the body nullable?
This the current contract requires me to have special knowledge about how to use EmptyContent
and do something like this:
CLIENT.request<String> {
url(website.pollUrl)
for (header in store.websiteExtractor.headers) {
header(header.key, header.value)
}
body = extractor.body?.injectVariables(website.data) ?: EmptyContent
}
when this requires no special knowledge and is easier:
body = extractor.body?.injectVariables(website.data)
Breaking change: formUrlEncode no longer encodes = in the parameters' values
To reproduce:
Evaluate the following code with Ktor 1.4.2
and Ktor 1.4.3
:
Parameters.build {
append("name", "val=ue")
}.formUrlEncode()
For 1.4.2
the return value will be: name=val%3Due
.
For 1.4.3
the return value will be: name=val=ue
.
Original discussing in slack https://kotlinlang.slack.com/archives/C0A974TJ9/p1606852506259100?thread_ts=1606852506.259100&cid=C0A974TJ9
ktor-client-js - Cannot read property Json of undefined
Hi! I'm one of pretty happy users of Kotlin and Ktor. Our game community decided to use Ktor and KVision as a frameworks for our web site and API and we also use Ktor HTTP client in both JVM and JS parts.
Today I was upgrading some of the dependencies and after that I faced with scary error that breaks frontend and that I cannot understand. I mean, it's so scary that it doesn't reaches fun main()
:
Uncaught TypeError: Cannot read property 'Json_x26noe$' of undefined
at Object.eval (ktor-ktor-client-serialization-jsLegacy.js?34df:32)
at eval (ktor-ktor-client-serialization-jsLegacy.js?34df:3)
at eval (ktor-ktor-client-serialization-jsLegacy.js?34df:24)
at Object../kotlin-dce-dev/ktor-ktor-client-serialization-jsLegacy.js (main.bundle.js:3299)
at __webpack_require__ (main.bundle.js:30)
at eval (cec-platform-frontend.js:3)
at eval (cec-platform-frontend.js:8)
at Object../kotlin-dce-dev/cec-platform-frontend.js (main.bundle.js:3200)
at __webpack_require__ (main.bundle.js:30)
at Object.0 (main.bundle.js:3399)
I tried to search it on the internet and Kotlin Slack channels, but with no clues. Also, I have a feeling that this is a some Kotlin-js compiler issue, but not sure about that.
Will be very appreciate for help.
Versions:
Kotlin: 1.4.21
Gradle:
------------------------------------------------------------
Gradle 6.1.1
------------------------------------------------------------
Build time: 2020-01-24 22:30:24 UTC
Revision: a8c3750babb99d1894378073499d6716a1a1fa5d
Kotlin: 1.3.61
Groovy: 2.5.8
Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM: 11.0.8 (Oracle Corporation 11.0.8+10)
OS: Linux 5.3.18-lp152.57-default amd64
OS: openSUSE Leap 15.2 (Linux)
KVision: 3.17.2
Ktor: 1.4.3
If you need, I can give a link to a project on the GitHub. Also, I'll try to reproduce issue in minimal environment tomorrow.
iOS client fails with CoroutinesInternalError when Logging is used
Creating a simple HttpClient
with the Logging
feature included fails with the attached stack trace. The failure occurs both for real web calls in-app, as well as for a unit test using MockEngine
. Both the test and the in-app call succeed if the logging configuration is removed.
I have a simple reproducer project attached, and also available at https://github.com/russhwolf/Ktor14
Note that my test configuration is using version 1.3.9-native-mt
of kotlinx.coroutines
, because version 1.3.9
leads to KTOR-915 / KTOR-919
kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[EventLoopImpl@36817bc8, Continuation @ $split$lambda-0$<anonymous>_3COROUTINE$2]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlin.Error#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:14)
at kotlinx.coroutines.CoroutinesInternalError#<init>(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Exceptions.common.kt:28)
at kotlinx.coroutines.DispatchedTask#handleFatalException(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask#run(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:64)
at kotlinx.coroutines.EventLoopImplBase#processNextEvent(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:274)
at kotlinx.coroutines#runEventLoop(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:80)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:67)
at kotlinx.coroutines#runBlocking(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:50)
at com.example#runBlocking(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:8)
at com.example#runBlocking$default(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:7)
at com.example.ApiClientTest#mockWebCall(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:20)
at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.invoke#internal(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.$<bridge-UNNN>invoke(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
at kotlin.native.internal.test.BaseClassSuite.TestCase#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestSuite.kt:85)
at kotlin.native.internal.test.TestRunner.run#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:236)
at kotlin.native.internal.test.TestRunner.runIteration#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:258)
at kotlin.native.internal.test.TestRunner#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:273)
at kotlin.native.internal.test#testLauncherEntryPoint(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:22)
at kotlin.native.internal.test#main(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:26)
at <global>.Konan_start(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:25)
at <global>.Init_and_run_start(Unknown Source)
at <global>.start(Unknown Source)
at kotlin.Exception#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:25)
at kotlin.RuntimeException#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:36)
at kotlinx.coroutines.CompletionHandlerException#<init>(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Exceptions.common.kt:13)
at kotlinx.coroutines.JobSupport.notifyCompletion#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:365)
at kotlinx.coroutines.JobSupport.completeStateFinalization#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:326)
at kotlinx.coroutines.JobSupport.finalizeFinishingState#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:241)
at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:922)
at kotlinx.coroutines.JobSupport.tryMakeCompleting#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:875)
at kotlinx.coroutines.JobSupport#makeCompletingOnce(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:840)
at kotlinx.coroutines.AbstractCoroutine#resumeWith(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/AbstractCoroutine.kt:111)
at kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/coroutines/ContinuationImpl.kt:43)
at kotlinx.coroutines.DispatchedTask#run(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:53)
at kotlinx.coroutines.EventLoopImplBase#processNextEvent(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:274)
at kotlin.Throwable#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23)
at kotlin.Exception#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23)
at kotlin.RuntimeException#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34)
at kotlin.native.concurrent.InvalidMutabilityException#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:22)
at <global>.ThrowInvalidMutabilityException(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:92)
at <global>.MutationCheck(Unknown Source)
at kotlinx.coroutines.AwaitAll.AwaitAllNode.<set-handle>#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:94)
at kotlinx.coroutines.AwaitAll.await#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:68)
at kotlinx.coroutines.$awaitAllCOROUTINE$1#invokeSuspend(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:38)
at kotlinx.coroutines#awaitAll@kotlin.collections.Collection<kotlinx.coroutines.Deferred<0:0>>(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:37)
at io.ktor.util.$split$lambda-0COROUTINE$3.invokeSuspend#internal(/opt/buildAgent/work/a85294440dc5c6e/ktor-utils/common/src/io/ktor/util/ByteChannels.kt:29)
at kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/coroutines/ContinuationImpl.kt:30)
at kotlinx.coroutines.DispatchedTask#run(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:39)
at kotlinx.coroutines.EventLoopImplBase#processNextEvent(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:274)
at kotlinx.coroutines#runEventLoop(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:80)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:67)
at kotlinx.coroutines#runBlocking(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:50)
at com.example#runBlocking(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:8)
at com.example#runBlocking$default(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:7)
at com.example.ApiClientTest#mockWebCall(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:20)
at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.invoke#internal(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.$<bridge-UNNN>invoke(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
at kotlin.native.internal.test.BaseClassSuite.TestCase#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestSuite.kt:85)
at kotlin.native.internal.test.TestRunner.run#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:236)
at kotlin.native.internal.test.TestRunner.runIteration#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:258)
at kotlin.native.internal.test.TestRunner#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:273)
at kotlin.native.internal.test#testLauncherEntryPoint(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:22)
at kotlin.native.internal.test#main(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:26)
at <global>.Konan_start(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:25)
at <global>.Init_and_run_start(Unknown Source)
ktor-client-apache: thread stuck in ByteBufferChannel.readRemainingSuspend
In production I've got two threads stuck in 100% CPU usage due to Ktor.
The server is running without debugging agent, so I cannot debug the exact scenario just yet. I'll restart it today with debugging agent running so that I can investigate further the next time the issue occurs.
When creating thread dumps I get one of the following two outputs. Note the high CPU usage and that the threads seem to be stuck in ByteBufferChannel.readRemainingSuspend
.
"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51203112.18ms elapsed=53179.31s tid=0x00007f6fc0052510 nid=0xa9b2 runnable [0x00007f6f0ebe9000]
java.lang.Thread.State: RUNNABLE
at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible$default(ByteBufferChannel.kt:2678)
at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:2231)
at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51253887.85ms elapsed=53230.30s tid=0x00007f6fc0052510 nid=0xa9b2 runnable [0x00007f6f0ebe9000]
java.lang.Thread.State: RUNNABLE
at io.ktor.utils.io.core.internal.UnsafeKt.prepareWriteHead(Unsafe.kt:160)
at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:3060)
at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Core
Significant performance penalty with socket write() and file read() on JVM
Hi,
It seems that there are performance issues with the current implementation of TCP socket write and also with reading files. Attaching the performance comparison for ktor IO vs regular JVM raw IO:
Files:
--------------------------------------------------------------------
Ktor channel read (128MiB), 20 iterations
Avg: 1239.65ms Min: 1095ms, Max: 1723ms
Raw: 1095, 1104, 1113, 1113, 1113, 1122, 1123, 1126, 1151, 1184, 1249, 1251, 1254, 1254, 1278, 1314, 1351, 1399, 1476, 1723
--------------------------------------------------------------------
Random File Read (128MiB), 20 iterations
Avg: 75.4ms Min: 65ms, Max: 161ms
Raw: 65, 65, 66, 67, 67, 67, 67, 68, 68, 68, 68, 69, 69, 70, 70, 75, 79, 83, 96, 161
--------------------------------------------------------------------
Stream Read (128MiB), 20 iterations
Avg: 58.5ms Min: 56ms, Max: 69ms
Raw: 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 59, 59, 59, 59, 59, 65, 69
Sockets:
--------------------------------------------------------------------
JVM socket write (128MiB), 20 iterations
Avg: 87.7ms Min: 79ms, Max: 112ms
Raw: 79, 81, 81, 82, 82, 82, 83, 83, 84, 84, 84, 86, 87, 90, 90, 91, 92, 96, 105, 112
--------------------------------------------------------------------
Ktor socket write (128MiB), 20 iterations
Avg: 675.3ms Min: 572ms, Max: 968ms
Raw: 572, 582, 583, 589, 598, 612, 633, 636, 651, 676, 682, 683, 686, 688, 689, 702, 727, 763, 786, 968
You can reproduce the performance measurements with the following tests:
https://gist.github.com/Malinskiy/2dbe05a38321417fd4f23e1872593806
https://gist.github.com/Malinskiy/0172b369040cabc074ccfea9a787a79c
This is the result of an investigation of the issue raised here https://github.com/Malinskiy/marathon/issues/462
Cheers,
Anton
ByteBufferChannel.readRemaining suspend when should not
Reproducer:
suspend fun main() {
val channel = ByteChannel(true)
println(channel.availableForRead) //prints 0
channel.writePacket {
writeText("hello")
}
println(channel.availableForRead) //prints 5
val packet = channel.readRemaining(channel.availableForRead.toLong()) //suspend...
// val packet = channel.readPacket(channel.availableForRead)
println(packet.readText())
}
Channel has 5 bytes available for read.
Contract of availableForRead
says, that:
/**
* Returns number of bytes that can be read without suspension. Read operations do no suspend and return
* immediately when this number is at least the number of bytes requested for read.
*/
public actual val availableForRead: Int
But in readRemaining
it suspends on trying to read...
Using readPacket
it works ok.
Using ByteChannelSequential on JVM don't suspend on both calls. On K/N the same - both don't suspends.
Looks like it regression after KTOR-1268. Caused by this change: https://github.com/ktorio/ktor/pull/2214/files#diff-408be3eaf4a8ec196449c0483c1525a24450590d9473ebcc3a5bc3798326e3e7L2234-R2222
Now channel is trying to read one byte, even if it's not needed.
Reproduced on:
kotlin: 1.4.32
ktor: 1.5.3
Fix socket self connect problem
I already made a pr on github: https://github.com/ktorio/ktor/pull/2237
Subsystem
ktor-network-jvm
Motivation
TCP has a well known self-connect problem, which client can connect to the client itself without any program listen on the port.
Reproduce Code :
fun testSelfConnect() {
runBlocking {
// Find a port that would be used as a local address.
val port = getAvailablePort()
val tcpSocketBuilder = aSocket(ActorSelectorManager(Dispatchers.IO)).tcp()
// Try to connect to that address repeatedly.
for (i in 0 until 100000) {
try {
val socket = tcpSocketBuilder.connect(InetSocketAddress("127.0.0.1", port))
println("connect to self succeed: ${socket.localAddress} to ${socket.remoteAddress}")
System.`in`.read()
break
} catch (ex: Exception) {
// ignore
}
}
}
}
// since new linux kernel version introduce a feature, new bind port number will always be odd number
// and connect port will always be even, so we find a random even port with while loop
// see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=07f4c90062f8fc7c8c26f8f95324cbe8fa3145a5
private fun getAvailablePort(): Int {
while (true) {
val port = ServerSocket().apply {
bind(InetSocketAddress("127.0.0.1", 0))
close()
}.localPort
if (port % 2 == 0) {
return port
}
try {
// try bind the next even port
ServerSocket().apply {
bind(InetSocketAddress("127.0.0.1", port + 1))
close()
}
return port + 1
} catch (ex: Exception) {
// ignore
}
}
}
Running the above code whill result in a tcp connection with src addr equal to dst addr and src port equal to dst port.
If this happens, client will behave abnormal because it connect to the wrong server,server will bind fail on this machine.
Other programming language also have this problem, eg golang: https://golang.org/issue/2690
Fix this problem is easy, just check if the local and remote socket pair are equal.
Incorrect encoding function used for URL path by URLBuilder
If you use URLBuilder
to create a URL, it incorrectly encodes the path. Per https://tools.ietf.org/html/rfc3986#section-2 3.3, a The path is terminated by the first question mark ("?") or number sign ("#") character, or by the end of the URI.
. However the #
symbol isn't encoded by the path
function.
val rawPath = "id+test&test~test#test"
val urlString = url {
path(rawPath)
}
// urlString = http://localhost/id+test&test~test#test
If you look at the source of the path
function you will see it uses <string>.encodeURLQueryComponent()
which is the incorrect function. If it used the <string>.encodeURLPath()
the #
would be correctly encoded as id+test&test~test%23test
.
public fun path(components: List<String>): URLBuilder {
encodedPath = components.joinToString("/", prefix = "/") { it.encodeURLQueryComponent() }
return this
}
Input.readTextExactBytes(n) on empty input different behavior per platform
On JVM, if Input
is empty, it will throw EOFException
On Native/JS it will just return empty string
reproducer:
println(ByteReadPacket.Empty.readTextExactBytes(1).length)
JVM: fail with EOF
JS/Native: print 0
ktor-io: JVM shared function decrease performance starting from 1.4.0
While profiling rsocket-kotlin
I found in graphs/traces that one of the most frequent element are calls to SharedJVM
and boxing of Int
values. I know, that shared
delegate used in ktor-io
to support using atomics on K/N and plain value on JVM/JS. But that code optimization
isn't ok regarding to performance.
The issue consists of two parts:
- every usage of
shared
- create anonymous object on initialization - boxing/unboxing of primitives on setting/getting value
F.e.Buffer
class fromktor-io
useshared
for 4Int
fields:readPosition
,writePosition
,startGap
,limit
- and all of them are updating frequently. So all time we operate on them we will do box/unbox - performance suffer significantly...
To proof my idea, I've replaced calls of shared
to plain variables inside ktor-io
module, and built it locally. Running benchmarks I found, that it's 50% faster!
So, f.e. io.rsocket.kotlin.benchmarks.RSocketBenchmark#requestResponseParallel
benchmark on ktor version 1.4.2
gives ~200 ops/s and on local version without shared gives ~300 ops/s.
That benchmark heavily use big ByteReadPacket
s.
I think that should be fixed some how, as 50% difference is really big
Experimental API and compatibility guarantees
Problem
Currently, we have a public commitment to not break any API and ABI. This is generally good.
The problem is that it makes it almost impossible for a new experimental feature/API to grow and entirely stops evolution since major versions are released too slowly.
Proposed solution
The idea is to relax guarantees for all API that is marked with experimental annotations similar to Kotlin.
Rules
- in a patch version: no changes allowed, deprecations with level WARNING could be introduced for existing API.
- in a minor version: both source and binary changes allowed except for complete removal. All changes should include migration/deprecation. Hidden deprecated APIs should be introduced for binary compatibility.
- in a major version: any changes including complete removal but only through the deprecation cycle.
Requirements for experimental API:
- experimental API should be marked with an experimental annotation requiring opt-in
- an experimental annotation should have level ERROR and should have a reasonable and clear description
- for a single or group of new experimental functions/classes/APIs having the same purpose and/or related functionality, it is required to introduce a new experimental annotation
Note: the consequence is that you can't use KtorExperimentalAPI
annotation for new API. So one should always create new experimental annotations for now.
The other consequence is that we should revisit all KtorExperimentalAPI usages and either stabilize them or migrate to new experimental annotations.
Requirements for experimental API annotation:
An experimental annotation class should:
- require opt-in with level ERROR (
@RequiresOptIn(level = ERROR)
); - have a reasonable and clear description;
- have a reference to a ktor ticket that introduced this API and/or describes why the API is experimental and that is useful for tracking the concrete experimental feature status;
- have retention
BINARY
, unless there is a reason to haveRUNTIME
; - be documented;
- contain a disclamer in the kdoc about the risk of using experimental features.
Note: RUNTIME
retention is not recommended since the ability to analyze it in runtime makes it more difficult to remove the annotation due to indirect and unobvious references. Retention SOURCE
is not acceptable because it is how @RequiresOptIn
works in Kotlin.
Experimental API Deprecation cycle
A deprecation cycle for an API that is intended to be removed or changed consist of the following steps:
- In any release, make the API deprecated with level WARNING by applying one of the following:
@Deprecated
annotation with level WARNING, provide a clear message describing what will be changed and what exact migration is required, provide a "replace with"- a new annotation requiring opt-in with level WARNING and the description
- In the next minor release, promote the deprecation level from WARNING to ERROR
- In the next minor release after step 2, change the deprecation level from ERROR to HIDDEN, keep the implementation working despite that it is hidden (to keep binary compatibility)
- In the next major release, remove the old API
Please note that using an opt-in annotation for deprecation doesn't provide the ability to write a "replace with". Thus, it only makes sense to use it for removing old API. A good example is kotlinx.coroutines ObsoleteCoroutinesApi
If some of the rules are impossible to comply (for example, it is impossible to provide a "replace with" action or impossible to keep a hidden function working), an author of change should initiate a discussion at least with PR reviewer. When decided, it could be also discussed in the team or even publically for a quite popular experimental API. If it was decided to skip some of the requirements, it is important to document that decision and reasons in the PR.
KtorExperimentalAPI deprecation
@KtorExperimentalAPI
annotation is going to be deprecated in the following stages:
- Revisit all existing API marked with the annotation and stabilize when possible without breaking changes in a minor version(s). Mark the annotation deprecated with level WARNING.
- In major version (2.0.0), revisit again. When impossible to stabilize, introduce new experimental annotations and use them instead. Ensure no KtorExperimentalAPI usages. Promote its deprecation to level ERROR.
- In the next major version (3.0.0) deprecate it with level HIDDEN. Use text search to ensure no text occurrences.
- In the next major version (4.0.0) remove it.
The reason why we deprecate it is that it introduces a way of "anonymous" experiment. An experimental API should be introduced with a clear reason and plan. A plan should cover only one particular feature/functionality. An experimental feature should have a more or less clear lifecycle.
The other disadvantage of this annotation is that there is no easy way to de-opt-in users from it. Also, it invites most brave users to opt-in globally for the whole project (just as we did for ExperimentalCoroutinesApi
). This is very bad because in this case experimental usages will become uncontrolled and the user doesn't realize anymore what danger it is. So to prevent such problems, we should stop using it and migrate to a better approach.
Example
Here is an example of an experimental API:
/**
* API marked with this annotation is a part of experimental Z feature.
* The feature preview provides the ability to work with Z.
* Please see KTOR-number for details and stabilization plan.
*
* This is an API preview and it's not recommended for production development
* since it could be changed or removed in the future and
* not guaranteed to remain functional and its behaviour may also change in any non-patch release.
*/
@RequiresOptIn("Z feature is experimental and not recommended for production development.", level = ERROR)
@Retention(BINARY)
annotation class ZFeature
/**
* Z feature's context
*/
@ZFeature
class ZContext
/**
* Configures Z feature and invokes [block] with running instance.
*/
@ZFeature
fun withZ(block: () -> Unit) = ...
Docs
java.net.MalformedURLException: unknown protocol: ws KTOR android
as stated in documentation I want to create websocket in Server and connect to it from Android my code for Android:
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val client = HttpClient {
install(WebSockets)
}
lifecycleScope.launchWhenCreated {
withContext(Dispatchers.IO) {
client.ws(
method = HttpMethod.Get,
host = "ws://192.168.43.4",
port = 23569, path = ""
) { // this: DefaultClientWebSocketSession
// Send text frame.
send("Hello, Text frame")
// Send text frame.
send(Frame.Text("Hello World"))
// Receive frame.
val frame = incoming.receive()
when (frame) {
is Frame.Text -> println(frame.readText())
is Frame.Binary -> println(frame.readBytes())
}
}
}
}
}
}
but it gives the following error:
java.net.MalformedURLException: unknown protocol: ws
at java.net.URL.<init>(URL.java:597)
at java.net.URL.<init>(URL.java:487)
at java.net.URL.<init>(URL.java:436)
at io.ktor.client.engine.android.AndroidClientEngine.getProxyAwareConnection(AndroidClientEngine.kt:102)
at io.ktor.client.engine.android.AndroidClientEngine.execute(AndroidClientEngine.kt:47)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
OAuth authentication documentation is confusing because of incomplete code example
Example in this documentation page contains calls of extension functions: loginFailedPage
, loggedInSuccessResponse
and loginPage
which aren't defined. So a user is confused and cannot use code from that example easily.
Also, the Locations
feature is used there without an installation block and the link to its documentation is missing.
Post from the forum https://discuss.kotlinlang.org/t/trying-to-get-ktor-examples-to-run/12296.
Full example for configuring response validation is missing
Duplicate entry "Features" in Server docs
See Server -> Developing Applications
, there are two entries titled "Features": an article about features and a group of articles about the core features.
And they have the same URL https://ktor.io/docs/features.html.
Missing dependency information in the Timeout topic
Web feedback from "https://ktor.io/docs/nopage.html"
Name says about https://ktor.io/docs/nopage.html:
Testing feedback service
Guides topic is empty
This topic https://ktor.io/docs/guides.html has no content.
No content for the Metrics feature topic
The content of this page https://ktor.io/docs/feature-metrics.html is empty.
Apache client engine is documented to support HTTP/2 but doesn't
https://ktor.io/docs/http-client-engines.html#apache
Apache is the most configurable HTTP client about right now. It supports HTTP/1.1 and HTTP/2.
Unfortunately Ktor uses httpclient 4.4 but HTTP/2 support requires httpclient 5.
https://archive.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
https://hc.apache.org/httpcomponents-client-4.4.x/index.html
Server features list is empty
Here https://ktor.io/docs/zfeatures.html#features we should have a features list or some other content but this section is empty.
Missing dependency information the Client Auth topic
Add information about required artifacts to the WebSockets topic
Outdated dependency information in the Client Logging topic
Center the docs
It would be great if the docs were centered. This change should be easy to implement and would make a huge difference on large monitors.
In my opinion, it would make the docs look way more professional.
(If you love the docs being on the left, then give the viewer an option, just like with the dark mode - but in reality I think the majority wants centered docs)
Thanks in advance!
Generator
Ktor generator fails to generate when Auth-Oauth is selected
If yo select Oauth feature it will fail to generate project either with a plugin or with a website.
The problem is that it requires Locations feature but it is not aligned with all-features.json
DoubleReceive and Kotlinx.serialization features are missing in generator
Infrastructure
settings.gradle: if (native_targets_enabled) does nothing
The native targets are always include, ignoring the variable. Looks like a wrong merge :)
https://github.com/ktorio/ktor/settings.gradle#L54
if (native_targets_enabled) {
include ':ktor-client:ktor-client-curl'
include ':ktor-client:ktor-client-ios'
}
include ':ktor-client:ktor-client-curl'
include ':ktor-client:ktor-client-ios'
Add build parameter to build ktor with JVM IR compiler
From PR #2199:
To be used in specific kotlinx-train CI configuration. It's needed for faster adoption of JVM IR compiler.
CI does not trigger build on PR to samples repo
PRs to https://github.com/ktorio/ktor-samples are not checked by CI
Invalid publication 'kotlinMultiplatform' for task :publishKotlinMultiplatformPublicationToMavenLocal on master
To reproduce run ./gradlew :publishKotlinMultiplatformPublicationToMavenLocal
in the root directory of the Ktor framework repository using the latest master.
> Failed to publish publication 'kotlinMultiplatform' to repository 'mavenLocal'
> Invalid publication 'kotlinMultiplatform': multiple artifacts with the identical extension and classifier ('jar', 'null').
OS: macOS Catalina
IntelliJ IDEA Plugin
Ktor plugin - A new project with Digest authentication uses deprecated API
- Create a new project in IDEA.
- Choose
Authentication Digest
ACT: A project uses the deprecateduserNameRealmPasswordDigestProvider
property.
EXP: A project usesdigestProvider
.
Server
Memory leak on Https request
Hi I try to use Ktor server running on Android.
The embedded server is selected to be Netty.
When it handles TLS connections. I notice the memory usage keeps higher while handling https requests
After inpecting via the profiling tool on Android studio, the native heap is not going low.
What the SSL handler used is Conscrypt (which is included with "org.conscrypt:conscrypt-android:2.5.1"), and I add provider "Security.insertProviderAt(Conscrypt.newProvider(), 1);" before running Ktor server
It works well when it handles non-https connections where memory usage is not going higher
Is there any solution for this? Thanks!
Android OS: 8
Unexpected exception when using Session feature: "Using blocking primitives on this dispatcher is not allowed"
kotlin: 1.4.20
kotlinx.coroutines: 1.4.2
Sessions feature is configured in a rather standard manner:
install(Sessions) {
cookie<UserIdPrincipal>(
Cookies.SESSION_ID,
storage = directorySessionStorage(sessionStorageRootDir)
) {
cookie.path = "/"
cookie.httpOnly = true
cookie.extensions["SameSite"] = "Lax"
}
}
When it is called during a pipeline execution, an unexpected exception is thrown.
java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
(Coroutine boundary) ~[?:?]
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:259) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) ~[ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) ~[ktor-server-netty-jvm-1.4.3.jar:1.4.3]
Caused by: java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.4.20.jar:1.4.20-release-308 (1.4.20)]
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:342) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:27) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:109) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.async(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.async$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:37) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:31) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_251]
at io.ktor.sessions.BaseCache.getOrCompute(Cache.kt:36) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache.getOrCompute(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseTimeoutCache.getOrCompute(Cache.kt:163) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:63) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.launch(Unknown Source) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:43) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:34) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Server losing channel exceptions at receive
If the input channel is broken due to some reason (broken connection or wrong transfer encoding parameters), the server may lose the exception and receive an empty request body and continue working.
Jackson: Crash when sending large responses in 1.4.2
Hi there,
We've discovered that Ktor 1.4.2 appears to crash out when sending large responses, using the Jackson feature. I think it might be related to KTOR-1342, but it's also happening on sending. Downgrading to ktor-jackson:1.4.1
seems to fix the issue.
The error logged is:
Exception in thread "eventLoopGroupProxy-3-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@2b5e16e4, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@114e04cb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
I've created a sample project for you to try - https://github.com/SkyeWelch/ktor-1342-error-sample . You can run it with ./gradlew run
, and if it's the same as on my machine see the following responses (I'm using httpie to make requests):
→ http localhost:8080/small
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked
{
"string": "small string"
}
→ http localhost:8080/large
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked
http: error: ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Please let me know if you need anything else.
Thanks,
Skye
GMTDate: Operator overload (plus/minus) for kotlin.time.Duration
As Ktor already has some overloads for kotlin.time.Duration
in ktor-server-core/features/KotlinTimeJvm.kt
, this overload would be nice too.
val date = GMTDate() + 1.hours
/**
* Adds the specified [duration]
*/
@ExperimentalTime
public operator fun GMTDate.plus(duration: Duration): GMTDate = GMTDate(timestamp + duration.toLongMilliseconds())
/**
* Subtracts the specified [duration]
*/
@ExperimentalTime
public operator fun GMTDate.minus(duration: Duration): GMTDate = GMTDate(timestamp - duration.toLongMilliseconds())
Request headers exceeding expected threshold are not handled correctly
Kotlin version: 1.3.70
Ktor version: 1.3.2
In situations when receiving large cookies and the headers exceed the 8K limit (using Netty as engine), not all headers get loaded.
Example 1:
- When we have the
ContentNegotiation
feature enabled - And a
content-type: application/json
is present on the request - And the cookie header is very large (the sum of all headers exceeding 8K)
- Then the
ContentNegotiation
feature resolves the content type toAny
(*/*
) - probably because it can't load all headers - This results in a situation where the server responds with
415 - Unsupported Media Type
.
In this particular example probably a 431 - Requests header fields too large
is more appropriate.
Example 2:
- Consider the hypothesis from Example 1
- When the
content-type: application/json
header is loaded before the cookie header - Then the request passes successfully through the
ContentNegotiation
feature - But the request hangs
This seems like an issue where Ktor does not correctly interpret errors from the Netty engine.
Sample engine error:
DefaultHttpRequest(decodeResult: failure(io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.), version: HTTP/1.1)
POST /v1/api HTTP/1.1
Host: localhost:8080
Accept: */*
cache-control: no-cache
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
content-type: text/plain;charset=UTF-8
Upgrade Netty to 4.1.54.Final
"Wrong HEX escape": gracefully handle invalid URLs
We recently got some alerts in our webserver because ktor threw an uncaught exception. Ideally these are handled and a 400 is sent back.
Here is the exception:
Wrong HEX escape: %uf, in /%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/winnt/win.ini
Stacktrace:
at io.ktor.http.CodecsKt.decodeImpl (Codecs.kt:197)
at io.ktor.http.CodecsKt.decodeScan (Codecs.kt:146)
at io.ktor.http.CodecsKt.decodeURLPart (Codecs.kt:140)
at io.ktor.http.CodecsKt.decodeURLPart$default (Codecs.kt:139)
at io.ktor.routing.RoutingResolveContext.parse (RoutingResolve.kt:70)
at io.ktor.routing.RoutingResolveContext.<init> (RoutingResolve.kt:49)
at io.ktor.routing.Routing.interceptor (Routing.kt:31)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend (Routing.kt:99)
at io.ktor.routing.Routing$Feature$install$1.invoke (Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend (ContentNegotiation.kt:107)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke (ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.util.pipeline.SuspendFunctionGun.execute (PipelineContext.kt:183)
at io.ktor.util.pipeline.Pipeline.execute (Pipeline.kt:27)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend (DefaultEnginePipeline.kt:120)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke (DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.util.pipeline.SuspendFunctionGun.execute (PipelineContext.kt:183)
at io.ktor.util.pipeline.Pipeline.execute (Pipeline.kt:27)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend (NettyApplicationCallHandler.kt:40)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke (NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched (Undispatched.kt:55)
at kotlinx.coroutines.CoroutineStart.invoke (CoroutineStart.kt:111)
at kotlinx.coroutines.AbstractCoroutine.start (AbstractCoroutine.kt:158)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch (Builders.common.kt:54)
at kotlinx.coroutines.BuildersKt.launch (Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest (NettyApplicationCallHandler.kt:30)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead (NettyApplicationCallHandler.kt:24)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600 (AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run (AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute (AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks (SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
Netty HTTP/2 HEAD response hangs
HttpServerTestSuite.testHeadResponse
fails with Netty HTTP/2 because a response message has no actual body but headers have the concrete content length so it leads to the pipeline failure
Sessions + SSL (Netty)
edit: https://github.com/dm6801/ktor_sessions_ssl
when trying to use Session feature with cookies
install(Sessions) {
cookie<MySession>("MY_SESSION")
}
accessing https:/localhost:443/ throws:
java.lang.UnsupportedOperationException: null
(Coroutine boundary)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:55)
(Coroutine creation stacktrace)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:188)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:38)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:821)
Caused by: java.lang.UnsupportedOperationException: null
at io.ktor.server.netty.http2.NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64)
at io.ktor.sessions.SessionTransportCookie.receive(SessionTransportCookie.kt:28)
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:204)
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:59)
at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
relevant line NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64):
override val cookies: RequestCookies
get() = throw UnsupportedOperationException()
- http://localhost:8080/ - works
Support for precompresed files
From PR #540
Nowadays, it is quite common to pre-compress static files ( a.k.a assets ) to reduce response time (no need for on the fly compression).
This pull request enables that, serving the pre-compressed version of the file, suppressing compression(if installed) and responding appropriate Content-Type and Content-Encoding. As brotli is not supported at the time of writing, it also covers serving brotli-compressed files.
Digest authentication: cannot successfully pass authentication using curl or web browser
To reproduce run the server using the following code snippet:
fun main() {
embeddedServer(Netty, port = 8081) {
install(Authentication) {
digest("digest") {
val password = "123"
realm = "testrealm@host.com"
digestProvider { name, realm ->
"$name:$realm:$password".toByteArray()
}
}
}
routing {
authenticate("digest") {
get("/digest") {
call.respondText("ok")
}
}
}
}.start(wait = true)
}
Send the following HTTP request using curl
:
curl -v 'http://localhost:8081/digest' --digest -u guest:123
I expect successfully pass authentication and get a response with a status code 200 but instead, I get the following response:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="43f48979ca35e2a5", algorithm="MD5"
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8081/digest'
* Found bundle for host localhost: 0x7fbe62518750 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> Authorization: Digest username="guest", realm="testrealm@host.com", nonce="43f48979ca35e2a5", uri="/digest", response="ee8c56f444996c77cd1e5f7570035372", algorithm="MD5"
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="76d13a440ec99b1c", algorithm="MD5"
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0
HTML Page response size above 4099 bytes causes fatal sever exceptions
I have an extremely weird issue with Ktor. The following exception is thrown after about 2-3 reloads of a page.
Exception in thread "eventLoopGroupProxy-3-2" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@259245dc, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@6200633a]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2262)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:909)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1952)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:307)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
... 22 more
However, I can reliably avoid the issue when I remove some of the HTML code that is being sent back for this request.
So basically; when I change a portion of my HTML output for a route to be shorter than it should be, the error will never occur.
As soon as I put back the HTML, basically crossing some limit, the Exception is thrown always after the third/fourth reload.
I have created the following repository that demonstrates the issue (entirey reproducible for me): https://github.com/ApoY2k/exception-demo
I also included two HAR files, one with a working request/response and one with a failing one (notice that the reponse does work, but the exception is still thrown which causes other server issues down the line)
For future reference, the main class code is this: (also attached is a package of the whole repository)
package apoy2k.exception.demo
import io.ktor.application.*
import io.ktor.response.*
import io.ktor.request.*
import io.ktor.routing.*
import io.ktor.http.*
import io.ktor.html.*
import kotlinx.html.*
fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
routing {
get("/") {
call.respondHtml {
// 4058 works
// 4059 throws exception
for (x in 1..4058) {
+"a"
}
}
}
}
}
It seems that as soon as the response HTML is larger than 4099 bytes it crashes. Anything until that limit runs perfectly fine.
What is going on here? How can I possibly fix this?
Jackson: Crash when sending large responses in 1.4.3
Hi,
It seems that this issue didnt fix on ktor 1.4.2 (see KTOR-1369) and fired similar exception on Ktor 1.4.3 using the Jackson feature. Downgrading to ktor-jackson:1.4.1 seems to fix the issue.
The error logged is:
Exception in thread "eventLoopGroupProxy-3-1 @response-pipeline#2" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@17d44fe5, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@335313af]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:144)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:115)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2262)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:909)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1952)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:307)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.internal.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.internal.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:104)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.internal.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.internal.DispatchedContinuation are in unnamed module of loader 'app')
... 22 more
Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
failing on: val bookRequest = call.receive<Example>()
(full example):
package app.controllers
import app.models.Example
import app.services.ExampleService
import io.ktor.application.*
import io.ktor.auth.*
import io.ktor.auth.jwt.*
import io.ktor.features.*
import io.ktor.http.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import org.kodein.di.instance
class ExampleController(application: Application): Controller(application) {
private val exampleService: ExampleService by instance()
override fun Route.getRoutes() {
post("/") {
val bookRequest = call.receive<Example>()
exampleService.addExample(bookRequest)
call.respond(HttpStatusCode.Accepted)
}
}
}
Parsing Authorization header throws IllegalArgumentException instead of returning null
Method io.ktor.http.auth.parseAuthorizationHeader(headerValue: String): HttpAuthHeader?
throws IllegalArgumentException instead of returning null in some cases. Looking at the source code, the problem occurs because validation that the method does is weaker than the validation HttpAuthHeader.Single
and HttpAuthHeader.Parametrized
do in their constructors. In effect, when header has non-token68 characters in schema, the method throws when instantiating HttpAuthHeader
.
To reproduce the issue, call the method with value, which includes non-token68 characters before first whitespace:
parseAuthorizationHeader("B<arer MyToken")
To fix the issue, the pattern checked for authScheme should discard values, which don't have token68 characters.
Other
Implement Support for Compression Extensions for WebSocket (RFC 7692)
Upgrade gradle to 6.7.1 and enable fs watcher
testWriteXRaceCondition sometimes hangs on CI
Remove copyTo usage from ServerPipeline
Fix failing tests in master after 1.5.0 merge
Handle failure in reading request body
@Test
fun testErrorInWritingPropagates() = testSuspend {
val client = HttpClient(factory)
val channel = ByteChannel(true)
channel.writeAvailable("text".toByteArray())
channel.close(IllegalStateException("Error on write"))
assertFailsWith<IllegalStateException>("Error on write") {
val result = client.post<String>("http://localhost:$serverPort/echo") {
body = channel
}
}
}
Apache: sends empty body without error
OkHttp: hangs
CIO: sends empty body without error
Android: succeed
`ByteBufferChannel.readRemaining` doesn't read whole channel
Client: URL encode / escaping is wrong
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1945
Ktor Version and Engine Used (client or server and name)
Version: 1.3.2-1.4-M1-2
ktor-client-core
ktor-client-json-native
ktor-client-serialization-native
ktor-client-curl
Describe the bug
The URL is wrong escaped.
To Reproduce
class FahrplanApi(private val serverUrl: String = "http://api.deutschebahn.com/freeplan/v1") {
private val client = HttpClient {
install(JsonFeature)
// workaround for version 1.4-M1
// https://github.com/ktorio/ktor/issues/1820#issuecomment-633519483
HttpResponseValidator {
validateResponse { response ->
when (response.status.value) {
in 300..399 -> throw RedirectResponseException(response)
in 400..499 -> throw ClientRequestException(response)
in 500..599 -> throw ServerResponseException(response)
}
}
}
expectSuccess = false
}
suspend fun departureBoardDateTime(id: Long, date: String) =
client.get<Departure>("$serverUrl/departureBoard/$id?date=$date")
}
fun main(): Unit = runBlocking {
val api = FahrplanApi()
api.departureBoardDateTime(8000096, "2020-06-14T20:21:22")
}
after run:
Uncaught Kotlin exception: io.ktor.client.features.ClientRequestException: Client request(https://api.deutschebahn.com/freeplan/v1/departureBoard/8000096?date=2020-06-14T20%253A21%253A22) invalid: 400 Bad Request
Expected behavior
https://api.deutschebahn.com/freeplan/v1/departureBoard/8000096?date=2020-06-14T20%3A21%3A22
%253A
--> %3A
(25 is actually %
)
The first call.receiveText() hangs until timeout
Ktor Server version
1.4.3, 1.5.2 (both affected)
Feedback
I’ve recently developed a simple Ktor app and organised a hosting for it on Apache Webserver + Tomcat.
The app has a post { … } route used for processing HTTP POST requests.
post("/api/v1/meeting") {
logger.trace { "Before call.receiveText()" }
val text = call.receiveText()
logger.trace { "After call.receiveText()" }
...
}
The route works fine for all requests except the first one (and sometimes 2 or 3). Additional tracing shows that request processing stucks on the line "call.receiveText()", where there is an attempt to read incoming JSON body of HTTP POST.
The request is stuck until network timeout.
Every following POST request with exactly the same content is processed fine. Assuming some lazy loading issue, I put “load-on-startup” in tomcat servlet setting - it didn’t affect the result.
Attaching the logs:
Apache access.log (you can see the timeout at the end)
Apache error.log (no clues)
Tomcat catalina logs (no clues)
Servlet log (only the first tracing message is in the log)
Tried to add ktor {.development=true } into application.conf, it didn't bring additional info.
P.S. the issue is reported following the JB request in https://stackoverflow.com/questions/66437014/ktor-first-call-receivetext-gets-timeout
Fix configuration if project without VPN and cache
Update kotlin to 1.4.20
Wrong pool is used to release `IOBuffer` after `ByteChannelSequential.copyTo` from static initialized instance.
How to create a response interceptor that reads the current response from server and changes it conditionally
I'm trying to implement a conditional response interceptor. It would be installed as a feature to my HttpClient
implementation and use the responsePipeline
to intercept server responses in my client application. But I cannot seem to implement it successfully. Ktor errors out with DoubleReceiveException
when I try to use readText()
on the HttpResponse
instance. I tried to replace the call to readText()
with the following code:
val byteArray = ByteArray(response.content.availableForRead)
response.content.readAvailable(byteArray)
return String(byteArray)
It seems to work in my interceptor, but then my code that's doing the API call fails with:
kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 0: Expected '{, kind: CLASS'
JSON input:
How can I process the ByteReadChannel
to String
without the aforementioned issues?
Basically, it is not clear what should I do. There are no samples available for this kind of use case. Any example would go a long way. Thank you.
Ktor post(any - request) with execute
Hey! I did not found how I might exclude using suspend when I use to execute? When I use to execute it async request, it not sync work and suspend we not need.
Not able to build iOS framework with Xcode 11.3.1 and ktor version 1.4.2 onwards. However, it works with ktor version 1.4.1
Attaching the error logs.
> Task :shared:linkDebugFrameworkIosArm64 FAILED
e: /Users/soumyajitdas/XCode Installer/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld invocation reported errors
The /Users/soumyajitdas/XCode Installer/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld command returned non-zero exit code: 1.
output:
Undefined symbols for architecture arm64:
"___darwin_check_fd_set_overflow", referenced from:
_io_ktor_network_interop_select_fd_add_wrapper1 in result.o
_io_ktor_network_interop_select_fd_isset_wrapper3 in result.o
ld: symbol(s) not found for architecture arm64
Execution failed for task ':shared:linkDebugFrameworkIosArm64'.
> Compilation finished with errors
1.4.2
released 10th November 2020
Client
Add WebSocket support for the iOS
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1894
Ktor Version and Engine Used (client or server and name)
Using io.ktor:ktor-client-ios:1.3.2
Describe the bug
This code prints kotlin.IllegalStateException: [generateNonce] is not supported on iOS
:
GlobalScope.launch {
try {
HttpClient {
install(WebSockets)
}.wss(
host = "websocketstest.com",
path = "/service"
) {
send(Frame.Text("timer"))
for (frame in incoming)
println((frame as Frame.Text).readText())
}
} catch (e: Exception) {
println(e)
}
}
Looks like someone tried to fix this here: https://github.com/ktorio/ktor/pull/1535
No transformation found: class io.ktor.utils.io.ByteBufferChannel -> class kotlin.collections.List
I`m using KTOR for create Http requests in KMM Project (android + ios)
My requests don't work at all
Occur exception: No transformation found: class io.ktor.utils.io.ByteBufferChannel -> class kotlin.collections.List
Configuration of my project
val serializationVersion = "1.0.0-RC"
val ktorVersion = "1.4.2"
val coroutinesVersion = "1.3.9-native-mt"
val sqlDelightVersion: String by project
val kotlinVersion = "1.4.0"
val koinVersion = "3.0.0-alpha-4"
sourceSets {
val commonMain by getting {
dependencies {
implementation("org.jetbrains.kotlin:kotlin-reflect:$kotlinVersion")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:$coroutinesVersion")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core-common:1.1.0")
implementation("org.jetbrains.kotlinx:kotlinx-serialization-core:1.0.0-RC")
implementation("io.ktor:ktor-client-cio:$ktorVersion")
implementation("io.ktor:ktor-client-core:$ktorVersion")
implementation("io.ktor:ktor-client-json:$ktorVersion")
implementation("io.ktor:ktor-client-logging:$ktorVersion")
implementation("io.ktor:ktor-client-serialization:$ktorVersion")
implementation("com.squareup.sqldelight:runtime:$sqlDelightVersion")
implementation("com.squareup.sqldelight:coroutines-extensions:$sqlDelightVersion")
implementation("org.koin:koin-core:$koinVersion")
}
}
val commonTest by getting {
dependencies {
implementation(kotlin("test-common"))
implementation(kotlin("test-annotations-common"))
}
}
val androidMain by getting {
dependencies {
implementation("io.ktor:ktor-client-android:$ktorVersion")
implementation("io.ktor:ktor-client-core:$ktorVersion")
implementation("io.ktor:ktor-client-json:$ktorVersion")
implementation("io.ktor:ktor-client-logging:$ktorVersion")
implementation("io.ktor:ktor-client-serialization:$ktorVersion")
implementation("io.ktor:ktor-client-gson:$ktorVersion")
implementation("io.ktor:ktor-client-okhttp:$ktorVersion")
implementation("com.squareup.sqldelight:android-driver:$sqlDelightVersion")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.3.9")
}
}
val androidTest by getting {
dependencies {
implementation(kotlin("test-junit"))
implementation("junit:junit:4.13")
}
}
val iosMain by getting {
dependencies {
implementation("io.ktor:ktor-client-ios:$ktorVersion")
implementation("com.squareup.sqldelight:native-driver:$sqlDelightVersion")
}
}
My Code
import io.ktor.client.*
import io.ktor.client.engine.okhttp.*
import io.ktor.client.features.*
import io.ktor.client.features.json.*
import io.ktor.client.features.json.serializer.*
import io.ktor.client.features.logging.*
import io.ktor.util.*
actual class Network actual constructor() {
@KtorExperimentalAPI
actual fun getHttpClient(): HttpClient {
return HttpClient(OkHttp) {
defaultRequest {
install(HttpTimeout) {
requestTimeoutMillis = 5000
connectTimeoutMillis = 3000
}
install(JsonFeature) {
serializer = KotlinxSerializer()
}
install(Logging) {
logger = Logger.ANDROID
level = LogLevel.ALL
}
}
}
}
}
My Object
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
@Serializable
data class OnboardingResponse(
@SerialName("title") val title: String,
@SerialName("subtitle") val subtitle: String,
@SerialName("description") val description: String,
@SerialName("image_url") val imageUrl: String
)
Http request
class OnboardingDataSource(
private val httpClient: HttpClient
) : IOnboardingDataSource {
private val mapper by lazy { OnboardingMapper() }
override suspend fun getList(): List<OnboardingData> {
return httpClient
.get<List<OnboardingResponse>>(
UrlProvider().getHostUrl().plus(UrlProvider.onboarding)
)
.map { mapper.transform(it) }
}
}
"IllegalStateException: Cannot execute task because event loop was shut down" for native client
To reproduce run ./gradlew runDebugExecutableMacosX64
in the root directory of the attached project.
As a result, I get the following unexpected exception:
Uncaught Kotlin exception: kotlin.IllegalStateException: Cannot execute task because event loop was shut down
at 0 loop_shutdown_coroutines.kexe 0x000000010b88270d kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23:37)
at 1 loop_shutdown_coroutines.kexe 0x000000010b87ba7b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44)
at 2 loop_shutdown_coroutines.kexe 0x000000010b87bc3b kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44)
at 3 loop_shutdown_coroutines.kexe 0x000000010b87c14b kfun:kotlin.IllegalStateException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:70:44)
at 4 loop_shutdown_coroutines.kexe 0x000000010ba5639e kfun:kotlinx.coroutines#loopWasShutDown(){}kotlin.Nothing + 222 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/CoroutineContext.kt:25:50)
at 5 loop_shutdown_coroutines.kexe 0x000000010ba5887d kfun:kotlinx.coroutines.EventLoopImplPlatform#reschedule(kotlin.Long;kotlinx.coroutines.EventLoopImplBase.DelayedTask){} + 61 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/EventLoop.kt:22:9)
at 6 loop_shutdown_coroutines.kexe 0x000000010b9e657b kfun:kotlinx.coroutines.EventLoopImplBase.rescheduleAllDelayed#internal + 587 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:397:13)
at 7 loop_shutdown_coroutines.kexe 0x000000010b9e2dde kfun:kotlinx.coroutines.EventLoopImplBase#shutdown(){} + 270 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:226:9)
at 8 loop_shutdown_coroutines.kexe 0x000000010ba58d1c kfun:kotlinx.coroutines.EventLoopImpl#shutdown(){} + 140 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/EventLoop.kt:34:15)
at 9 loop_shutdown_coroutines.kexe 0x000000010b9e12a4 kfun:kotlinx.coroutines.EventLoop#decrementUseCount(kotlin.Boolean){} + 324 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:114:13)
at 10 loop_shutdown_coroutines.kexe 0x000000010b9e137c kfun:kotlinx.coroutines.EventLoop#decrementUseCount$default(kotlin.Boolean;kotlin.Int){} + 124 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:108:5)
at 11 loop_shutdown_coroutines.kexe 0x000000010ba5383b kfun:kotlinx.coroutines#runEventLoop(kotlinx.coroutines.EventLoop?;kotlin.Function0<kotlin.Boolean>){} + 1115 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:85:20)
at 12 loop_shutdown_coroutines.kexe 0x000000010ba551f0 kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 464 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:67:9)
at 13 loop_shutdown_coroutines.kexe 0x000000010ba52dcf kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 1311 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:50:22)
at 14 loop_shutdown_coroutines.kexe 0x000000010ba5330c kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 348 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:33:15)
at 15 loop_shutdown_coroutines.kexe 0x000000010b82bc48 kfun:#main(){} + 168 (/Users/Aleksei.Tirman/projects/jb/loop_shutdown_coroutines/src/macosX64Main/kotlin/App.kt:6:14)
at 16 loop_shutdown_coroutines.kexe 0x000000010b82c5c0 Konan_start + 144 (/Users/Aleksei.Tirman/projects/jb/loop_shutdown_coroutines/src/macosX64Main/kotlin/App.kt:6:1)
at 17 loop_shutdown_coroutines.kexe 0x000000010b831f4b Init_and_run_start + 107
at 18 libdyld.dylib 0x00007fff6a1c9cc9 start + 1
Unfinished workers detected, 1 workers leaked!
Use `Platform.isMemoryLeakCheckerActive = false` to avoid this check.
Abort trap: 6
Here is the similar issue.
I've reproduced this problem for both macOS and Linux targets.
Kotlin multiplatform: 1.4.10
Kotlin coroutines: 1.3.9-native-mt-2
Ktor: 1.4.0
Ktor WebSocket with OkHttpClient authenticator
I have that client
private val client: HttpClient = HttpClient(OkHttp) {
engine {
preconfigured = okHttpClient
}
}.config {
install(WebSockets)
install(JsonFeature) { serializer = GsonSerializer() }
}
okHttpClient :
private fun okHttpClient(
loggingInterceptor: Interceptor,
tokenAuthenticator: Authenticator,
pingIntervalSec: Long = 20
) = with(OkHttpClient.Builder()) {
connectTimeout(CONNECTION_TIMEOUT, TimeUnit.SECONDS)
readTimeout(READ_TIMEOUT, TimeUnit.SECONDS)
pingInterval(pingIntervalSec, TimeUnit.SECONDS)
// if (BuildConfig.DEBUG)
addInterceptor(loggingInterceptor)
authenticator(tokenAuthenticator)
build()
}
Authenticator:
class AuthAccessTokenAuthenticator(
private val oAuthRepo: OAuthRepo
) : Authenticator {
override fun authenticate(route: Route?, response: Response): Request? {
return runBlocking {
val accessTokenData = oAuthRepo.getAuthAccessToken()
.let {
if (it is ResponseState.Success)
it.result
else
null
}
if (isRequestWithAccessToken(response)
|| accessTokenData == null)
return@runBlocking null
return@runBlocking newRequestWithAccessToken(
response.request,
accessTokenData.accessToken
)
}
}
private fun isRequestWithAccessToken(response: Response): Boolean {
val header = response.request.header("Authorization")
return header != null && header.startsWith("Bearer")
}
private fun newRequestWithAccessToken(request: Request, accessToken: String): Request {
return request.newBuilder()
.header("Authorization", "Bearer $accessToken")
.build()
}
}
You can see that i added authenticator to my OkHttpClient but client.ws doesn't add an authorization token to my requests and I have to manually add it via header (). Why?
client.ws(host = "host",
path = "path",
request = {
Timber.d("Sockets -> Building socket HttpRequest")
url.protocol = URLProtocol.WS
url.port = 80
Timber.d("Sockets -> Starting socket connection")
//workaround for authorization header("Authorization", "token")
})
``
ResponseException is no longer serializable starting from 1.4.0 (breaking change)
Previously a ResponseException could be serialized. See: https://github.com/ktorio/ktor/issues/1256. The Transient annotation was removed in a recent commit for the HttpResponse
field. Now an exception is thrown when serializing a ResponseException.
Stack trace:
io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
java.io.NotSerializableException: io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
at ExceptionsTest.serialize(ExceptionsTest.kt:26)
at ExceptionsTest.testResponseExceptionSerializable(ExceptionsTest.kt:17)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Here is a test to reproduce the issue:
import io.ktor.client.call.*
import io.ktor.client.features.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.util.date.*
import io.ktor.utils.io.*
import java.io.*
import kotlin.coroutines.*
import kotlin.test.*
class ExceptionsTest {
@Test
fun testResponseExceptionSerializable() {
val exception = createResponseException()
val serialized = serialize(exception)
val deserialized = deserialize(serialized)
deserialized as ResponseException
}
private fun serialize(obj: Any): ByteArray {
val b = ByteArrayOutputStream()
val o = ObjectOutputStream(b)
o.writeObject(obj)
return b.toByteArray()
}
private fun deserialize(bytes: ByteArray): Any? {
val b = ByteArrayInputStream(bytes)
val o = ObjectInputStream(b)
return o.readObject()
}
}
private fun createResponseException(): ResponseException = ResponseException(object : HttpResponse() {
override val call: HttpClientCall
get() = TODO("Not yet implemented")
override val status: HttpStatusCode
get() = TODO("Not yet implemented")
override val version: HttpProtocolVersion
get() = TODO("Not yet implemented")
override val requestTime: GMTDate
get() = TODO("Not yet implemented")
override val responseTime: GMTDate
get() = TODO("Not yet implemented")
override val content: ByteReadChannel
get() = TODO("Not yet implemented")
override val headers: Headers
get() = TODO("Not yet implemented")
override val coroutineContext: CoroutineContext
get() = TODO("Not yet implemented")
override fun toString(): String = "FakeCall"
}, cachedResponseText = "Fake text")
Ktor 1.4.1 fails to upload file (submit form) when logging level = LogLevel.BODY
I am using ktor 1.4.1 in my multiplatform project to submit a file.
the weird thing is that whenever I change the log level to BODY the form parameters are corrupt, but when it is INFO it works correctly.
install(Logging) {
logger = object : Logger {
override fun log(message: String) {
println(message)
}
}
level = LogLevel.BODY // FAILS to submit
level = LogLevel.INFO // OK
}
here is the code to submit file
httpClient.submitFormWithBinaryData(
"https://myendpoint",
formData {
appendInput(
key = "file",
headers = Headers.build {
append(ContentDisposition, " filename=video.mp4")
}
) { buildPacket { writeFully(data) } }
}
)
And when I tried to debug, I found out that when level = LogLevel.BODY
ktor does not send
Content-Type: multipart/form-data; boundary=XXX
in request. It is only available in logs but not actually sent. Maybe logger consumes it or sth like this happens.
Overload resolution ambiguity: HttpClient.ws
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.features.websocket.*
suspend fun main() {
HttpClient(CIO).ws("http://localhost") {}
}
Overload resolution ambiguity:
public suspend fun HttpClient.ws(urlString: String, request: HttpRequestBuilder.() -> Unit = ..., block: suspend DefaultClientWebSocketSession.() -> Unit): Unit defined in io.ktor.client.features.websocket
public suspend fun HttpClient.ws(urlString: String, request: HttpRequestBuilder.() -> Unit = ..., block: suspend DefaultClientWebSocketSession.() -> Unit): Unit defined in io.ktor.client.features.websocket
There's one in core-jvm
and one in cio-jvm
.
Content-type header is missing in Request when using LogLevel.BODY
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1690
Ktor Version and Engine Used (client or server and name)
version: 1.3.1
lib: ktor-client-cio
additional modules: ktor-client-logging-jvm
Describe the bug
Content-type header is missing when LogLevel.BODY
is set. (also I am using multi-part body, not sure if relevant). Header exists if removing logging or setting LogLevel.HEADER
.
To Reproduce
Steps to reproduce the behavior:
- Set logging
HttpClient {
install(Logging) {
logger = Logger.ANDROID
level = LogLevel.BODY
}
}
- Do multi-form request
val response = httpClient.post<String> {
url("http://someurl")
body = MultiPartFormDataContent(formData {
append(
"file",
"file.txt",
ContentType.parse("text/plain"),
contentBytes.size.toLong()
) {
writeFully(contentBytes)
}
})
- Observe missing
Content-Type: multipart/form-data; <...>
header
Expected behavior
Content-Type: multipart/form-data; <...>
in request exists
Jetty client engine with high concurrency randomly freezes, throws NPEs, ISEs or EofExceptions
Exception in thread "main" java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:222)
at org.eclipse.jetty.io.SelectorManager.connect(SelectorManager.java:193)
at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:409)
at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:382)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$connect$2.invoke(JettyHttpRequest.kt:61)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$connect$2.invoke(JettyHttpRequest.kt)
at io.ktor.client.engine.jetty.UtilsKt.withPromise(utils.kt:14)
at io.ktor.client.engine.jetty.JettyHttpRequestKt.connect(JettyHttpRequest.kt:59)
at io.ktor.client.engine.jetty.JettyHttpRequestKt.executeRequest(JettyHttpRequest.kt:31)
at io.ktor.client.engine.jetty.JettyHttp2Engine.execute(JettyHttp2Engine.kt:38)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Suppressed: java.lang.NullPointerException
... 17 more
Suppressed: java.lang.NullPointerException
... 17 more
Suppressed: java.lang.NullPointerException
... 17 more
Suppressed: java.lang.NullPointerException
... 17 more
import io.ktor.client.*
import io.ktor.client.engine.jetty.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.*
suspend fun main() = withContext(Dispatchers.Default) {
HttpClient(Jetty) {
expectSuccess = false
install(HttpTimeout) {
connectTimeoutMillis = 2_000
socketTimeoutMillis = 2_000
}
engine {
threadsCount = 200
}
}.use { client ->
coroutineScope {
repeat(5) {
repeat(200) {
launch {
client.get<HttpResponse>("https://esi.evetech.net/latest/markets/10000040/orders/?page=1")
.readText()
.let { println(it.length) }
}
}
}
}
}
println()
println("Done")
}
Native: ResponseObserver hangs coroutine indefinitely
Here's my reproducer:
launch {
try {
val client = HttpClient()
ResponseObserver.install(ResponseObserver {
// Whatever
println("Bonjour!")
}, client)
client.get<HttpResponse>("https://www.google.com/")
onSuccess()
} catch (t: Throwable) {
t.printStackTrace()
onError()
}
}
Reproducer project here: https://github.com/Ribesg/kotlin-reproducer/tree/ktor-1095
Just clone, open in XCode 11.x and Run (tested on simulator).
CertificatePinner buildErrorMessage outputs CertificatesInfo reference and variable name (HASH_ALGORITHM_SHA_256) instead of value
When making the certificate pinner fail on iOS, it outputs this:
Certificate pinning failure!
Peer certificate chain:
io.ktor.client.engine.ios.certificates.CertificatesInfo@3d83458.HASH_ALGORITHM_SHA_256fEbY8nKEFY2ODyxn2P0kyIq+C2/Rkke8ifByvsmamLw=: *.domain.com
io.ktor.client.engine.ios.certificates.CertificatesInfo@3d83458.HASH_ALGORITHM_SHA_256hETpgVvaLC0bvcGG3t0cuqiHvr4XyP2MTwCiqhgRWwU=: GlobalSign RSA OV SSL CA 2018
io.ktor.client.engine.ios.certificates.CertificatesInfo@3d83458.HASH_ALGORITHM_SHA_256cGuxAXyFXFkWm61cF4HPWX8S0srS9j0aSqN0k4AP+4A=: GlobalSign
It inserts the reference to the CertificatesInfo object followed by the name of the variable HASH_ALGORITHM_SHA_256.
Expected output can be read in the documentation of CertificatePinner.
This is a bug in the error message building in CertificatePinner.buildErrorMessage().
Client: InvalidCacheStateException when Vary header differs for 200 and 304 responses
I perform a lot of requests in parallel with relatively short-lived Expires values. That leads to plenty of exceptions like this:
Exception in thread "main" io.ktor.client.features.cache.InvalidCacheStateException: The entry for url: https://esi.evetech.net/latest/markets/10000035/orders/?page=1 was removed from cache
at io.ktor.client.features.cache.HttpCache$Companion$install$2.invokeSuspend(HttpCache.kt:104)
at io.ktor.client.features.cache.HttpCache$Companion$install$2.invoke(HttpCache.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
at io.ktor.client.HttpClient$1.invokeSuspend(HttpClient.kt:141)
at io.ktor.client.HttpClient$1.invoke(HttpClient.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:69)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
Ktor Client HTTP Cache for invalid or blank Expires header
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1892
Ktor Version and Engine Used (client or server and name)
1.3.2, client, io.ktor.client.features.cache.HttpCache
Describe the bug
If a server returns a blank or an invalid Expires HTTP header, the ktor client crashes if the feature HttpCache is enabled install(HttpCache)
To Reproduce
Steps to reproduce the behavior:
- Write the following:
val articleUrl = Url("https://www.nachrichten.at/panorama/weltspiegel/30-jahre-mauerfall-sinnbild-der-freiheit;art17,3184503")
val newsClient = HttpClient(Apache) {
install(HttpCache)
install(UserAgent) {
agent = "Mozilla/5.0"
}
}
try {
val call: HttpStatement = newsClient.get(articleSimilarityRequest.articleUrl)
val response: HttpResponse = call.receive()
val requestUrl = response.request.url.toString()
val rawHtml = response.readText()
println(rawHtml)
} catch (cause: Throwable) {
println("Unable to download article")
} finally {
newsClient.close()
}
- If you open the URL
https://www.nachrichten.at/panorama/weltspiegel/30-jahre-mauerfall-sinnbild-der-freiheit;art17,3184503
in the browser, you can see that the Expires response header is blank - An exception is thrown if you run the code:
2020-05-23 14:41:24.124 [DefaultDispatcher-worker-2 @request#48] ERROR a.s.api.util.service.DownloadService - Unable to download article
java.lang.IllegalStateException: Failed to parse date:
at io.ktor.http.DateUtilsKt.fromHttpToGmtDate(DateUtils.kt:40)
at io.ktor.client.features.cache.HttpCacheEntryKt.cacheExpires(HttpCacheEntry.kt:84)
at io.ktor.client.features.cache.HttpCacheEntryKt.HttpCacheEntry(HttpCacheEntry.kt:18)
at io.ktor.client.features.cache.HttpCacheEntryKt$HttpCacheEntry$1.invokeSuspend(HttpCacheEntry.kt)
(Coroutine boundary)
at io.ktor.client.HttpClient$1.invokeSuspend(HttpClient.kt:123)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:68)
at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:116)
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:79)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:89)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:34)
at io.ktor.client.HttpClient.execute(HttpClient.kt:164)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:43)
at app.shwoo.api.util.service.DownloadService.downloadArticle$suspendImpl(DownloadService.kt:101)
at app.shwoo.api.server.web.SimilarityResourceKt$similarity$1$4.invokeSuspend(SimilarityResource.kt:38)
at io.ktor.routing.Routing.executeResult(Routing.kt:147)
at io.ktor.routing.Routing.interceptor(Routing.kt:34)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:99)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:107)
at io.ktor.server.testing.TestApplicationEngine$callInterceptor$1.invokeSuspend(TestApplicationEngine.kt:292)
at io.ktor.server.testing.TestApplicationEngine$2.invokeSuspend(TestApplicationEngine.kt:50)
at io.ktor.server.testing.TestApplicationEngine$handleRequest$pipelineJob$1.invokeSuspend(TestApplicationEngine.kt:290)
Caused by: java.lang.IllegalStateException: Failed to parse date:
at io.ktor.http.DateUtilsKt.fromHttpToGmtDate(DateUtils.kt:40)
at io.ktor.client.features.cache.HttpCacheEntryKt.cacheExpires(HttpCacheEntry.kt:84)
at io.ktor.client.features.cache.HttpCacheEntryKt.HttpCacheEntry(HttpCacheEntry.kt:18)
at io.ktor.client.features.cache.HttpCacheEntryKt$HttpCacheEntry$1.invokeSuspend(HttpCacheEntry.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Expected behavior
It seems that the problem occurs in https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-core/common/src/io/ktor/client/features/cache/HttpCacheEntry.kt#L84. If the Expires header is blank an exception is thrown. The expected behavior is that the application handles the cache as expired if an invalid date is provided in the HTTP Expires header. Like described in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expires
NoTransformationFoundException when using JsonFeature and connecting through a proxy server
Steps of reproduction:
- Add windows or macOS targets to the Gradle build file if required.
- Run
./gradlew runDebugExecutableLinuxX64
(replace with your target)
As a result, the following exception is thrown:
> Task :runDebugExecutableLinuxX64 FAILED
Uncaught Kotlin exception: io.ktor.client.call.NoTransformationFoundException: No transformation found: class io.ktor.utils.io.ByteChannelNative -> class Mine
with response from https://httpbin.org/get:
status: 200 OK
response headers:
at kfun:kotlin.Throwable#<init>(){} (0x306ebe)
at kfun:kotlin.Exception#<init>(){} (0x3007b7)
at kfun:kotlin.RuntimeException#<init>(){} (0x300377)
at kfun:kotlin.UnsupportedOperationException#<init>(){} (0x300bf7)
at kfun:io.ktor.client.call.NoTransformationFoundException#<init>(io.ktor.client.statement.HttpResponse;kotlin.reflect.KClass<*>;kotlin.reflect.KClass<*>){} (0x579a53)
at kfun:io.ktor.client.call.HttpClientCall.$receiveCOROUTINE$12#invokeSuspend(kotlin.Result<kotlin.Any?>){}kotlin.Any? (0x578b21)
at kfun:io.ktor.client.call.HttpClientCall#receive(io.ktor.client.call.TypeInfo){}kotlin.Any (0x578fc6)
at kfun:$main$lambda-2COROUTINE$0.invokeSuspend#internal (0x640c1f)
at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x322d4f)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
Unfinished workers detected, 1 workers leaked!
Use `Platform.isMemoryLeakCheckerActive = false` to avoid this check.
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
at kfun:kotlinx.coroutines.DispatchedTask#run(){} (0x4114a6)
at kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long (0x3e9ace)
at kfun:kotlinx.coroutines#runEventLoop(kotlinx.coroutines.EventLoop?;kotlin.Function0<kotlin.Boolean>){} (0x423639)
at kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal (0x422de0)
at kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 (0x42223b)
at kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 (0x422741)
at kfun:#main(){} (0x63e8cc)
at Konan_start (0x641839)
at Init_and_run_start (0x66cdcb)
at __libc_start_main (0x7f0318cb20b3)
at (0x2b8029)
at ((nil))
Here is the related Github issue.
CIO may not work correctly with proxy for https requests
Steps:
- privoxy setup:
$ cat ./privoxyconfig
confdir /etc/privoxy
temporary-directory /tmp
logdir /var/log/privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action # Main actions file
actionsfile user.action # User customizations
filterfile default.filter
filterfile user.filter # User customizations
logfile privoxy.log
debug 1 # Log the destination for each request Privoxy let through. See also debug 1024.
debug 1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
debug 4096 # Startup banner and warnings
debug 8192 # Non-fatal errors
listen-address 0.0.0.0:8118
toggle 1
enable-remote-toggle 0
enable-remote-http-toggle 0
enable-edit-actions 0
enforce-blocks 1
buffer-limit 4096
enable-proxy-authentication-forwarding 0
trusted-cgi-referer http://www.example.org/
forwarded-connect-retries 0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
tolerate-pipelining 1
socket-timeout 300
$ cat ./Dockerfile
FROM alpine
RUN apk add --no-cache privoxy=3.0.28-r0
COPY privoxyconfig /etc/privoxy/config
EXPOSE 8118
CMD privoxy --no-daemon /etc/privoxy/config
$ docker build . -t privoxy
$ docker run --rm -p 127.0.0.1:8118:8118 privoxy
- set proxy as described in https://ktor.io/docs/proxy.html with
fun main(args: Array<String>) {
val client = HttpClient(CIO) {
install(UserAgent) {
agent = "Space Unfurl Bot"
}
install(ContentEncoding) {
gzip()
}
engine {
proxy = Proxy(Proxy.Type.HTTP, InetSocketAddress("127.0.0.1", 8118))
}
}
val r = runBlocking(CoroutineName("Get Metadata")) {
client.get<HttpStatement>("https://example.com").execute { call ->
"response code: ${call.status.description}"
}
}
println(r)
}
Request:
[X] https://example.com
CIO hangs for a long time and returns java.lang.IllegalArgumentException: Invalid TLS record type code: 72
[X] http://ya.ru
same result (ya.ru redirects to https
)
[V] http://example.com
returns page content
[V] Replace CIO with Apache - any requests work perfectly fine.
KTOR Default url (hostname)
Is there any possibility to define a hostname, or base url for all requests, so is not need to set the url on every request, avoiding possible errors.
class RestApi(private val httpClient: HttpClient) {
suspend fun loadUser(): GithubUserApiModel {
return httpClient.get("/user".buildHost())
}
private fun String.buildHost() = "https://${BuildConfig.GH_HOST}/api/v3${this}"
}
Apache HttpClient engine: connection exception does not propagate, receiving JobCancellationException
Following code:
try {
HttpClient(Apache).get<String> {
// Non existing domain
url.takeFrom("http:/googl.com")
}
catch (e: Exception) {
// e.cause is null, but should be instance of java.net.ConnectException
}
Should throw exception that has cause
with java.net.ConnectException
, which is thrown by Apache Http client.
Currently, exception is JobCancellationException with null cause
.
IncorrectDereferenceException thrown when ktor request call from other thread
Hello there, I see that this issue KTOR-499(https://youtrack.jetbrains.com/issue/KTOR-499) is mentioned in Ktor 1.4.1 Change log as fixed (https://github.com/ktorio/ktor/blob/master/CHANGELOG.md) and in the issue in the comments there is a statement that it is working now with the Ktor 1.4.0. I’ve forked the repository(https://github.com/RudolfHladik/ktor-mt-app), updated it to Ktor 1.4.1, Kotlin 1.4.10, Coroutines 1.3.9-native-mt and I get the IncorectDereferenceException . Is there something wrong I do or has the issue reapeared in Ktor 1.4.1?
Core
java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer when Ktor is built with JDK 9+
To reproduce:
- Run
./gradlew publishToMavenLocal
in the root directory of the Ktor framework repository on the latest master. Use JDK 9+ for a build. - Use the
1.4.3-SNAPSHOT
(at the time of writing) version of Ktor in a project with Maven local repository. - Build and run the following code with JDK 8:
suspend fun main(args: Array<String>) {
val client = HttpClient(CIO)
client.get<String>("http://example.com")
}
As the result unexpected exception will be thrown:
Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$3.invoke(ExceptionUtilsJvm.kt:106)
at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$3.invoke(ExceptionUtilsJvm.kt)
at io.ktor.utils.io.ExceptionUtilsJvmKt.tryCopyException(ExceptionUtilsJvm.kt:66)
at io.ktor.utils.io.ByteBufferChannelKt.rethrowClosed(ByteBufferChannel.kt:2606)
at io.ktor.utils.io.ByteBufferChannelKt.access$rethrowClosed(ByteBufferChannel.kt:1)
at io.ktor.utils.io.ByteBufferChannel.setupStateForWrite$ktor_io(ByteBufferChannel.kt:211)
at io.ktor.utils.io.ByteBufferChannel.copyDirect$ktor_io(ByteBufferChannel.kt:2758)
at io.ktor.utils.io.ByteReadChannelJVMKt.copyTo(ByteReadChannelJVM.kt:276)
at io.ktor.client.engine.cio.UtilsKt$withoutClosePropagation$2.invokeSuspend(utils.kt:173)
at io.ktor.client.engine.cio.UtilsKt$withoutClosePropagation$2.invoke(utils.kt)
at io.ktor.utils.io.CoroutinesKt$launchChannel$job$1.invokeSuspend(Coroutines.kt:129)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
at io.ktor.utils.io.ByteBufferChannel.prepareBuffer(ByteBufferChannel.kt:193)
at io.ktor.utils.io.ByteBufferChannel.setupStateForWrite$ktor_io(ByteBufferChannel.kt:242)
at io.ktor.utils.io.ByteBufferChannel.copyDirect$ktor_io(ByteBufferChannel.kt:2758)
at io.ktor.utils.io.ByteReadChannelJVMKt.copyTo(ByteReadChannelJVM.kt:276)
at io.ktor.utils.io.ByteReadChannelKt.copyAndClose(ByteReadChannel.kt:267)
at io.ktor.utils.io.ByteReadChannelKt.copyAndClose$default(ByteReadChannel.kt:266)
at io.ktor.network.sockets.TimeoutExceptionsCommonKt$mapEngineExceptions$2.invokeSuspend(TimeoutExceptionsCommon.kt:61)
at io.ktor.network.sockets.TimeoutExceptionsCommonKt$mapEngineExceptions$2.invoke(TimeoutExceptionsCommon.kt)
... 7 more
You can find a similar issue here.
Significant performance penalty with socket write() and file read() on JVM
Hi,
It seems that there are performance issues with the current implementation of TCP socket write and also with reading files. Attaching the performance comparison for ktor IO vs regular JVM raw IO:
Files:
--------------------------------------------------------------------
Ktor channel read (128MiB), 20 iterations
Avg: 1239.65ms Min: 1095ms, Max: 1723ms
Raw: 1095, 1104, 1113, 1113, 1113, 1122, 1123, 1126, 1151, 1184, 1249, 1251, 1254, 1254, 1278, 1314, 1351, 1399, 1476, 1723
--------------------------------------------------------------------
Random File Read (128MiB), 20 iterations
Avg: 75.4ms Min: 65ms, Max: 161ms
Raw: 65, 65, 66, 67, 67, 67, 67, 68, 68, 68, 68, 69, 69, 70, 70, 75, 79, 83, 96, 161
--------------------------------------------------------------------
Stream Read (128MiB), 20 iterations
Avg: 58.5ms Min: 56ms, Max: 69ms
Raw: 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 59, 59, 59, 59, 59, 65, 69
Sockets:
--------------------------------------------------------------------
JVM socket write (128MiB), 20 iterations
Avg: 87.7ms Min: 79ms, Max: 112ms
Raw: 79, 81, 81, 82, 82, 82, 83, 83, 84, 84, 84, 86, 87, 90, 90, 91, 92, 96, 105, 112
--------------------------------------------------------------------
Ktor socket write (128MiB), 20 iterations
Avg: 675.3ms Min: 572ms, Max: 968ms
Raw: 572, 582, 583, 589, 598, 612, 633, 636, 651, 676, 682, 683, 686, 688, 689, 702, 727, 763, 786, 968
You can reproduce the performance measurements with the following tests:
https://gist.github.com/Malinskiy/2dbe05a38321417fd4f23e1872593806
https://gist.github.com/Malinskiy/0172b369040cabc074ccfea9a787a79c
This is the result of an investigation of the issue raised here https://github.com/Malinskiy/marathon/issues/462
Cheers,
Anton
URLBuilder: Move Default Values to build() function
For implementing a baseURL feature PR (KTOR-730, KTOR-759, KTOR-699), it is not possible to distinguish between the relative urlString/sub
and the explicit, overriding the given baseURLString http://localhost/sub
, because both different urlStrings resolves to the same URLBuilder (http://localhost/sub/
).
Motivation:
Make all parameters of URLBuilder
optional and move the default values into the build()
function. This allows you inside the URLBuilder to decide, if the requested URL is only a relative URL or an explicit urlString.
// See [PR](https://github.com/ktorio/ktor/pull/2212) for `defaultRequest`
@Test
@Ignore
fun overriddenLocalhostWithPath() {
val localhost = HttpRequestBuilder().apply {
url.takeFrom("http://localhost/sub")
}.defaultRequest().build().url
assertEquals("http://localhost/sub", localhost.toString())
}
ktor-io: JVM shared function decrease performance starting from 1.4.0
While profiling rsocket-kotlin
I found in graphs/traces that one of the most frequent element are calls to SharedJVM
and boxing of Int
values. I know, that shared
delegate used in ktor-io
to support using atomics on K/N and plain value on JVM/JS. But that code optimization
isn't ok regarding to performance.
The issue consists of two parts:
- every usage of
shared
- create anonymous object on initialization - boxing/unboxing of primitives on setting/getting value
F.e.Buffer
class fromktor-io
useshared
for 4Int
fields:readPosition
,writePosition
,startGap
,limit
- and all of them are updating frequently. So all time we operate on them we will do box/unbox - performance suffer significantly...
To proof my idea, I've replaced calls of shared
to plain variables inside ktor-io
module, and built it locally. Running benchmarks I found, that it's 50% faster!
So, f.e. io.rsocket.kotlin.benchmarks.RSocketBenchmark#requestResponseParallel
benchmark on ktor version 1.4.2
gives ~200 ops/s and on local version without shared gives ~300 ops/s.
That benchmark heavily use big ByteReadPacket
s.
I think that should be fixed some how, as 50% difference is really big
Dependency for ktor-network 1.4.1 and 1.4.0 doesn't work on Kotlin/Multiplatform
I tried to apply the ktor-network dependency to my commonMain source set dependencies, but I received an error saying that the dependency was not found. I also tried to apply it to the dependencies of the sourceSet commonMain and it seems to work there.
Repositories applied: jcenter, mavenCentral
build.gradle.kts:
plugins {
kotlin("multiplatform") version "1.4.10"
kotlin("plugin.serialization") version "1.4.10"
}
val ktorVersion = "1.4.1"
val coroutinesVersion = "1.4.1"
val serializationVersion = "1.0.1"
allprojects {
plugins.apply("org.jetbrains.kotlin.multiplatform")
plugins.apply("org.jetbrains.kotlin.plugin.serialization")
repositories {
mavenCentral()
jcenter()
}
kotlin {
jvm {
compilations.all {
kotlinOptions.jvmTarget = "1.8"
}
testRuns["test"].executionTask.configure {
useJUnitPlatform()
}
}
val hostOs = System.getProperty("os.name")
val isMingwX64 = hostOs.startsWith("Windows")
val nativeTarget = when {
hostOs == "Mac OS X" -> macosX64("native")
hostOs == "Linux" -> linuxX64("native")
isMingwX64 -> mingwX64("native")
else -> throw GradleException("Host OS is not supported in Kotlin/Native.")
}
sourceSets {
val commonMain by getting {
dependencies {
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:$serializationVersion")
implementation("io.ktor:ktor-network:$ktorVersion")
implementation("io.ktor:ktor-io:$ktorVersion")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:$coroutinesVersion")
}
}
val commonTest by getting {
dependencies {
implementation(kotlin("test-common"))
implementation(kotlin("test-annotations-common"))
}
}
val jvmMain by getting
val jvmTest by getting {
dependencies {
implementation(kotlin("test-junit5"))
implementation("org.junit.jupiter:junit-jupiter-api:5.6.0")
runtimeOnly("org.junit.jupiter:junit-jupiter-engine:5.6.0")
}
}
val nativeMain by getting
val nativeTest by getting
}
}
}
error screenshot:
CIO client engine randomly runs into infinite loop under high load
Stuck in while (!closed)
in io.ktor.network.selector.ActorSelectorManager.process()
.
Probably related to KTOR-717.
Likely a duplicate of KTOR-531.
I also randomly get this:
Exception in thread "main" java.net.BindException: Can't assign requested address
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:503)
at java.base/sun.nio.ch.Net.connect(Net.java:492)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:751)
at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:31)
at io.ktor.network.sockets.ConnectUtilsJvmKt.connect(ConnectUtilsJvm.kt:19)
at io.ktor.network.sockets.TcpSocketBuilder.connect(TcpSocketBuilder.kt:38)
at io.ktor.client.engine.cio.ConnectionFactory.connect(ConnectionFactory.kt:24)
at io.ktor.client.engine.cio.Endpoint$connect$$inlined$repeat$lambda$1.invokeSuspend(Endpoint.kt:159)
at io.ktor.client.engine.cio.Endpoint$connect$$inlined$repeat$lambda$1.invoke(Endpoint.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:120)
at kotlinx.coroutines.TimeoutKt.withTimeoutOrNull(Timeout.kt:83)
at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:167)
at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Repro:
import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import kotlin.random.*
import kotlin.time.*
import kotlinx.coroutines.*
@OptIn(ExperimentalTime::class)
suspend fun main() = withContext(Dispatchers.Default) {
startServer()
delay(1_000)
val batchSize = 100
val threadCount = 1
HttpClient(CIO) {
engine {
threadsCount = threadCount
}
}.use { client ->
repeat(Int.MAX_VALUE) { i ->
val time = measureTime {
coroutineScope {
repeat(batchSize) {
launch {
client.get<HttpResponse>("http://localhost:8080").readText()
}
}
}
}
println("${i + 1}: $time")
}
}
}
private fun startServer() {
embeddedServer(Netty, port = 8080) {
val random = Random(0)
val data = buildString(100_000) { repeat(100_000) { append(random.nextInt(0, 10)) } }
routing {
get("/") {
call.respondText(data, ContentType.Text.Plain)
}
}
}.start()
}
Docs
The kotlinx.serialization feature documentation is using deprecated API
On page https://ktor.io/docs/serialization-converter.html
Block:
install(ContentNegotiation) {
serialization()
}
has deprecated code.
[docs] Wrong shadow plugin version in Fat JAR docs
The fatjar documentation page https://ktor.io/docs/fatjar.html#fat-jar-gradle recommends using shadow plugin 2.0.4.
But I faced this error:
Task :shadowJar FAILED
A problem was found with the configuration of task ':shadowJar' (type 'ShadowJar').
No value has been specified for property 'mainClassName'.
I fixed this by updating the plugin from 2.0.4 to 6.1.0 (latest)
Example project: https://github.com/egorklimov/ktor-fatjar-issue
Generator
"Using userNameRealmPasswordDigestProvider is an error" compilation error of a generated project
I created the project using https://start.ktor.io/#dependency=ktor-client-core&dependency=ktor-client-apache&dependency=ktor-client-cio&dependency=ktor-client-mock&dependency=ktor-client-http-timeout&dependency=ktor-client-auth-jvm&dependency=ktor-client-json-jvm&dependency=ktor-client-logging&dependency=ktor-client-user-agent&dependency=html-dsl&dependency=css-dsl&dependency=static-content&dependency=auth-basic&dependency=auth-digest&dependency=auth-jwt&dependency=auth-oauth&dependency=auth&dependency=ktor-jackson&dependency=ktor-locations&dependency=ktor-metrics&dependency=ktor-sessions&dependency=compression&dependency=call-logging&dependency=cors&dependency=data-conversion&dependency=forwarded-header-support&dependency=status-pages&dependency=routing&dependency=webjars&dependency=content-negotiation&dependency=https-redirect&dependency=partial-content&ktor-engine=tomcat&artifact-group=com.helpchoice&artifact-name=startTest
and imported it into IDEA. Once IDE finished indexing it shows the error on userNameRealmPasswordDigestProvider. The error is:
e: /Users/cab/Documents/Projects/hosting/GWSjr 2/src/Application.kt: (77, 13): Using 'userNameRealmPasswordDigestProvider: suspend (userName: String, realm: String) -> ByteArray?' is an error. Use digestProvider { } function instead.
Generator frontend: "TypeError: (0 , _components.default) is not a function" when non supported language specified in markdown
For example:
```application.install(Routing) {
....
The name of the language here is application.install(Routing) {
because it goes right after triple backticks.
The problem is that in file ktor-generator-frontend/node_modules/markdown-it-prism/build/index.js
function which loads languages required like this:
var _components = _interopRequireDefault(require("prismjs/components/"));
The react-scripts
uses webpack 4.44.2
that will wrongly resolve the above path to node_modules/prismjs/components.js
instead of node_modules/prismjs/components/index.js
because of a bug. Since file components.js
does exist, require
will return the exported object from that file. We cannot call an object as a function, therefore TypeError
.
Fix height of the right panel on the new generator website.
Currently the panel enhances its size as new features are being added. But I believe the height of the right panel should equal to the max height of the whole window.
Ktor generator produces erroneous build.gradle.kts if KTS is chosen
testImplementation "io.ktor:ktor-server-tests:$ktor_version"
: braces are missingimplementation("io.ktor:ktor-server-core:$ktor_version)")
: odd closing brace
Infrastructure
Fix project configuration in IDEA on mingw host
ktor-server-test-host indirectly depends on logback-classic
Using ktor-server-test-host
results in logback classic being pulled in as a test dependancy, the relevent dependancy tree is below.
+--- io.ktor:ktor-server-test-host -> 1.3.2
| +--- io.ktor:ktor-client-tests-jvm:1.3.2
| | +--- ch.qos.logback:logback-classic:1.2.3
| | | +--- ch.qos.logback:logback-core:1.2.3
| | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30
Add artifact dependency to release configuration
CI does not trigger build on PR to samples repo
PRs to https://github.com/ktorio/ktor-samples are not checked by CI
Invalid publication 'kotlinMultiplatform' for task :publishKotlinMultiplatformPublicationToMavenLocal on master
To reproduce run ./gradlew :publishKotlinMultiplatformPublicationToMavenLocal
in the root directory of the Ktor framework repository using the latest master.
> Failed to publish publication 'kotlinMultiplatform' to repository 'mavenLocal'
> Invalid publication 'kotlinMultiplatform': multiple artifacts with the identical extension and classifier ('jar', 'null').
OS: macOS Catalina
IntelliJ IDEA Plugin
Plugin: "Could not find io.ktor:ktor-client-http-timeout" when building project created with http-timeout feature
Steps:
- Install Ktor Plugin
- Create Project With the Wizzard, like in the screenshot
- Build/Run
Expectation: Application should build an run:
Build does not work, because the ktor-client-http-timeout dependency can not be found.
I checked it on the server: https://kotlin.bintray.com/ktor/io/ktor/. I can not find this dependency there.
Plugin: the Ktor project type tooltip in the New Project wizard contains text used for Java projects
Steps:
- Open IntelliJ IDEA 2020.3 EAP (IU-203.3645.34)
- Click New Project on the Welcome screen.
- Hover the mouse pointer over Ktor.
ACT: A tooltip contains the text for a Java project.
EXP: A tooltip contains Ktor-specific text, for example: 'Create a Ktor project with a specific set of features'.
Test compilation errors when project created via plugin with "Mock HttpClient Engine"
folders as created by the ktor idea plugin:
/
.gradle/
resources/
src/
test/
expected folders
/
.gradle/
src/
main/
kotlin/
java/ -- if i would have java src
resources/
test/
kotlin/
java/ -- if i would have java src
resources/
More details here: https://docs.gradle.org/current/userguide/organizing_gradle_projects.html#sec:separate_language_source_files
There's a secondary issue with the generated sources: the test file has compilation errors.
I've attached multiple pictures, along with the generated project.
No changelog for 1.4.0 and 1.4.1 on the plugin page in IDE Marketplace
Samples
Samples: native-client compilation fails on compileKotlinMetadata
To reproduce run ./gradlew :native-client:compileKotlinMetadata
> Task :native-client:compileKotlinMetadata
e: /Users/Aleksei.Tirman/projects/jb/ktor-samples/generic/samples/native-client/src/commonMain/kotlin/runner.kt: (1, 8): Unresolved reference: io
e: /Users/Aleksei.Tirman/projects/jb/ktor-samples/generic/samples/native-client/src/commonMain/kotlin/runner.kt: (2, 8): Unresolved reference: io
e: /Users/Aleksei.Tirman/projects/jb/ktor-samples/generic/samples/native-client/src/commonMain/kotlin/runner.kt: (3, 8): Unresolved reference: io
e: /Users/Aleksei.Tirman/projects/jb/ktor-samples/generic/samples/native-client/src/commonMain/kotlin/runner.kt: (5, 31): Unresolved reference: HttpClientEngine
e: /Users/Aleksei.Tirman/projects/jb/ktor-samples/generic/samples/native-client/src/commonMain/kotlin/runner.kt: (6, 18): Unresolved reference: HttpClient
Server
Unexpected exception when using Session feature: "Using blocking primitives on this dispatcher is not allowed"
kotlin: 1.4.20
kotlinx.coroutines: 1.4.2
Sessions feature is configured in a rather standard manner:
install(Sessions) {
cookie<UserIdPrincipal>(
Cookies.SESSION_ID,
storage = directorySessionStorage(sessionStorageRootDir)
) {
cookie.path = "/"
cookie.httpOnly = true
cookie.extensions["SameSite"] = "Lax"
}
}
When it is called during a pipeline execution, an unexpected exception is thrown.
java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
(Coroutine boundary) ~[?:?]
at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:259) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) ~[ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) ~[ktor-server-netty-jvm-1.4.3.jar:1.4.3]
Caused by: java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.4.20.jar:1.4.20-release-308 (1.4.20)]
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:342) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:27) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:109) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.async(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.async$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:37) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:31) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_251]
at io.ktor.sessions.BaseCache.getOrCompute(Cache.kt:36) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.ReferenceCache.getOrCompute(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.BaseTimeoutCache.getOrCompute(Cache.kt:163) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:63) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at kotlinx.coroutines.BuildersKt.launch(Unknown Source) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:43) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:34) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Jackson: Crash when sending large responses in 1.4.2
Hi there,
We've discovered that Ktor 1.4.2 appears to crash out when sending large responses, using the Jackson feature. I think it might be related to KTOR-1342, but it's also happening on sending. Downgrading to ktor-jackson:1.4.1
seems to fix the issue.
The error logged is:
Exception in thread "eventLoopGroupProxy-3-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@2b5e16e4, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@114e04cb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
I've created a sample project for you to try - https://github.com/SkyeWelch/ktor-1342-error-sample . You can run it with ./gradlew run
, and if it's the same as on my machine see the following responses (I'm using httpie to make requests):
→ http localhost:8080/small
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked
{
"string": "small string"
}
→ http localhost:8080/large
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked
http: error: ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Please let me know if you need anything else.
Thanks,
Skye
Micrometer metrics: scan of random URLs of a server exhausts available values per tag in influx
of("route", call.attributes[measureKey].route ?: call.request.path()),
This line here (https://github.com/ktorio/ktor/blob/master/ktor-features/ktor-metrics-micrometer/jvm/src/io/ktor/metrics/micrometer/MicrometerMetrics.kt#L125) should not assign arbitrary value to a tag.
The result of such behaviour is that if the application is maliciously scanned for every kind of URL possible, the application itself correctly returns 404 on such scans, but the metrics library still reports latency on every single one of the scans with its own tag value. This is very hard, for example, for influx, which has default limit on values per tag (https://stackoverflow.com/questions/43770354/max-values-per-tag-limit-exceeded-influxdb).
And overall, storage of such information is useless.
So there should be some sane limit on possible values of route
tag, possible limited by routes available in application.
"Wrong HEX escape": gracefully handle invalid URLs
We recently got some alerts in our webserver because ktor threw an uncaught exception. Ideally these are handled and a 400 is sent back.
Here is the exception:
Wrong HEX escape: %uf, in /%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/winnt/win.ini
Stacktrace:
at io.ktor.http.CodecsKt.decodeImpl (Codecs.kt:197)
at io.ktor.http.CodecsKt.decodeScan (Codecs.kt:146)
at io.ktor.http.CodecsKt.decodeURLPart (Codecs.kt:140)
at io.ktor.http.CodecsKt.decodeURLPart$default (Codecs.kt:139)
at io.ktor.routing.RoutingResolveContext.parse (RoutingResolve.kt:70)
at io.ktor.routing.RoutingResolveContext.<init> (RoutingResolve.kt:49)
at io.ktor.routing.Routing.interceptor (Routing.kt:31)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend (Routing.kt:99)
at io.ktor.routing.Routing$Feature$install$1.invoke (Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend (ContentNegotiation.kt:107)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke (ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.util.pipeline.SuspendFunctionGun.execute (PipelineContext.kt:183)
at io.ktor.util.pipeline.Pipeline.execute (Pipeline.kt:27)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend (DefaultEnginePipeline.kt:120)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke (DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.util.pipeline.SuspendFunctionGun.execute (PipelineContext.kt:183)
at io.ktor.util.pipeline.Pipeline.execute (Pipeline.kt:27)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend (NettyApplicationCallHandler.kt:40)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke (NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched (Undispatched.kt:55)
at kotlinx.coroutines.CoroutineStart.invoke (CoroutineStart.kt:111)
at kotlinx.coroutines.AbstractCoroutine.start (AbstractCoroutine.kt:158)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch (Builders.common.kt:54)
at kotlinx.coroutines.BuildersKt.launch (Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest (NettyApplicationCallHandler.kt:30)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead (NettyApplicationCallHandler.kt:24)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600 (AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run (AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute (AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks (SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
Netty HTTP/2 HEAD response hangs
HttpServerTestSuite.testHeadResponse
fails with Netty HTTP/2 because a response message has no actual body but headers have the concrete content length so it leads to the pipeline failure
Sessions + SSL (Netty)
edit: https://github.com/dm6801/ktor_sessions_ssl
when trying to use Session feature with cookies
install(Sessions) {
cookie<MySession>("MY_SESSION")
}
accessing https:/localhost:443/ throws:
java.lang.UnsupportedOperationException: null
(Coroutine boundary)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:55)
(Coroutine creation stacktrace)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:188)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:38)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:821)
Caused by: java.lang.UnsupportedOperationException: null
at io.ktor.server.netty.http2.NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64)
at io.ktor.sessions.SessionTransportCookie.receive(SessionTransportCookie.kt:28)
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:204)
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:59)
at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
relevant line NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64):
override val cookies: RequestCookies
get() = throw UnsupportedOperationException()
- http://localhost:8080/ - works
Sensible defaults for compression feature
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/544
At the moment Ktor is trying to compress even the steel.
I added this method in my core lib to overcome that.
I think there should be a similar thing active by default. What do you reckon?
fun Application.installDefaultCompresion() {
install(Compression) {
gzip {
matchContentType(
ContentType.Text.Any,
ContentType.Application.JavaScript,
ContentType.Application.Json,
ContentType.Application.Rss,
ContentType.Application.Xml,
ContentType.Application.Xml_Dtd,
ContentType.Application.Atom,
ContentType.Image.SVG,
ContentType.Image.XIcon
)
minimumSize(1400) // not worth compressing
}
}
}
Setting multiple cache control directives is impossible with current API
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/557
Using the install for CachingHeaders I want to configure it in such a way it produces this header:
Cache-Control: no-cache, no-store, must-revalidate
I cannot do it.
For example I tried this:
options { outgoingContent -> when (outgoingContent.contentType?.withoutParameters()) { ContentType.Text.CSS -> CachingOptions(CacheControl.NoCache(null)) else -> null } } options { outgoingContent -> when (outgoingContent.contentType?.withoutParameters()) { ContentType.Text.CSS -> CachingOptions(CacheControl.NoStore(null)) else -> null } }
it makes it twice:
cache-control: no-cache
cache-control: no-store
The best would be if options can return a lamda with a list of CachingOptions as return value, then the system would add all of them. Or allow caching options to accept varargs?
Also, why is "must revalidate" tied to max age?
See this, it can be also on it's own:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#Examples
Netty: NumberFormatException when make a request via HTTP/2 without explicit port
Hello.
There is a bug with Http2LocalConnectionPoint. If there is no port specified in authority then it tries to parse domain as port.
I've created a project to reproduce the bug: https://github.com/Vlad8161/netty-bug-sample
To reproduce it run the server on 443 port and access it without explicit port specifying.
Stack Trace:
java.lang.NumberFormatException: For input string: "localhost"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at io.ktor.server.netty.http2.Http2LocalConnectionPoint.getPort(Http2LocalConnectionPoint.kt:31)
at io.ktor.features.HSTS.intercept(HSTS.kt:87)
at io.ktor.features.HSTS$Feature$install$1.invokeSuspend(HSTS.kt:101)
at io.ktor.features.HSTS$Feature$install$1.invoke(HSTS.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:101)
at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:189)
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:100)
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:140)
at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139)
at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.startCoroutineImpl(Builders.common.kt:182)
at kotlinx.coroutines.BuildersKt.startCoroutineImpl(Unknown Source)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:145)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:54)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
The problem is at Http2LocalConnectionPoint:31
I think is would be better to do something like this:
nettyHeaders.authority()?.toString()?.substringAfter(":", if (scheme == "https") "443" else "80")?.toInt()
because often local port information is not enough
Thanks in advance :)
Class cast exception on web-socket cancelation
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1929
Ktor Version and Engine Used (client or server and name)
ktor-cio server 1.3.2
Describe the bug
I am getting an internal error when force-closing server with opened web-socket connection
Here is the stack trace:
13:32:20.839 [DefaultDispatcher-worker-4] ERROR io.ktor.server.cio.HttpServer - Unhandled exception caught for CoroutineName(http-pipeline-writer)
kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[ExperimentalCoroutineDispatcher@45e6d6a4[scheduler = DefaultDispatcher@6341301e[Pool Size {core = 10, max = 1024}, Worker States {CPU = 6, blocking = 0, parked = 8, dormant = 0, terminated = 0}, running workers queues = [0c, 0c, 0c, 0c, 0c, 0c], global CPU queue size = 5, global blocking queue size = 0, Control State {created workers= 14, blocking tasks = 0, CPUs acquired = 6}]], Continuation at io.ktor.server.cio.backend.ServerPipelineKt$startServerConnectionPipeline$1$outputsActor$1.invokeSuspend(ServerPipeline.kt:49)@2482cf33]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
... 4 common frames omitted
CORS - request with non-simple content type not failing
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1906
Ktor Version and Engine Used
Ktor 1.3.2, Netty
Here is the failing test:
@Test
fun testRequestWithNonSimpleContentTypeShouldFail() {
withTestApplication {
application.install(CORS) {
anyHost()
}
application.routing {
post("/") {
call.respond(HttpStatusCode.OK)
}
}
handleRequest(HttpMethod.Post, "/") {
addHeader(HttpHeaders.Origin, "http://my-host")
addHeader(HttpHeaders.ContentType, "application/json")
setBody("""{"cors":"test"}""")
}.apply {
assertEquals(HttpStatusCode.Forbidden, response.status())
}
}
}
I've read it that CORS feature doesn’t allow non-simple request body content types by default but this test fails. Am I understanding it wrong?
Scheme instead of version in OriginConnectionPoint
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1869
Ktor Version and Engine Used (client or server and name)
1.3.2, server
Describe the bug
ApplicationCall.request.origin.remoteHost
holds the scheme instead of the version (like 'http' instead of HTTP/1.1
).
The root cause is this line.
I provided a fix here: https://github.com/ktorio/ktor/pull/1871
To Reproduce
Steps to reproduce the behavior:
- Configure request logging for a ktor server
- Install 'XForwardedHeaderSupport`
- print
call.request.origin.remoteHost
- Run the server behind a proxy
- Send a request to the proxy
- Observe that
http
is printed
Expected behavior
HTTP/1.1
should be printed
Weird Ktor internal exception
Hello!
Exception in thread "eventLoopGroupProxy-3-2" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@1398676, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@2a2565a2]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException
This exception regularly appears in the output of my Ktor Webserver. Most of the time its harmless, but every 24 hours or so, it seems to be turning my webserver unresponsive (The requests arrive, but nothing gets sent back apparently), which forces me to restart the program. I am using version 8.0.282 of Azul Zulu OpenJDK, version 1.4.10 of Kotlin and version 1.4.2 of Ktor on Ubuntu 18.04. Help would be very much appreciated.
Best,
Darki
Netty: Thymeleaf java.lang.ClassCastException when template file is more than 4088 bytes in size
- Created a project via start.ktor.io
- replaced the content of the index.html file ( project to reproduce )
./gradlew run
and open browserhttp://0.0.0.0:8080/html-thymeleaf
- got:
Exception in thread "eventLoopGroupProxy-3-1 @response-pipeline#4" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@4d744708, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@600c2262]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
... 22 more
Fatal exception in ktor
I'm not sure why or where this happens, but we attempted to upgrade ktor to 1.4.2 and we got this error, I've downgraded to 1.3.2 which seems to have fixed it. I have two different stacktraces.
kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@6ed39d43, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@dc068af]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core (DispatchedTask.kt:144)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:115)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch (Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith (DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith (Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:106)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch (Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith (DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith (CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp (ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead (ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed (ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend (NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:106)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute (AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks (SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run (NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
at java.lang.Thread.run (Thread.java:834)
otlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@6ed39d43, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@2f25fce0]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core (DispatchedTask.kt:144)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:115)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch (Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith (DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith (Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:106)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch (Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith (DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith (CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp (ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead (ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed (ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend (NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:106)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute (AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks (SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run (NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
at java.lang.Thread.run (Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.internal.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.internal.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation (CoroutineDispatcher.kt:104)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted (ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:104)
NoSuchMethodError: No virtual method toPath() in android API < 26
Library:
io.ktor:ktor-server-core:1.4.2
Describe the bug
When I tried to serve files in a web server inside of an android app, when in API < 26, the following error is thrown:
java.lang.NoSuchMethodError: No virtual method toPath()Ljava/nio/file/Path; in class Ljava/io/File; or its super classes (declaration of 'java.io.File' appears in /system/framework/core-oj.jar)
at io.ktor.http.content.LocalFileContent.<init>(LocalFileContent.kt:27)
at io.ktor.http.content.LocalFileContent.<init>(LocalFileContent.kt:21)
at io.ktor.http.content.StaticContentKt$file$1.invokeSuspend(StaticContent.kt:73)
at io.ktor.http.content.StaticContentKt$file$1.invoke(StaticContent.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
...
After further investigation, I discovered that the class LocalFileContent uses java.nio.*, which is only supported in API >= 26, from what I've read.
This looks like it should have an easy enough fix, and until it is applied, I sadly cannot work with Ktor :(
To Reproduce
routing {
static {
staticRootFolder = filesDir
file("/", "index.html")
}
}
file() function uses LocalFileContent class
Fatal exception returning payloads via ContentNegotiation feature in JVM/Netty
- Cannot recreate in 1.4.1 or earlier versions
- Running on JVM (AdoptOpenJDK 11.0.7+10)
- Using Netty-based application engine
- Does not appear to be specifically related to
JacksonConverter
orContentNegotiation
code, but in how that work is handed off to threads
When using the JacksonConverter
with the ContentNegotiation
feature, payloads 4kb or larger throw an exception and ultimately return incomplete data (manifesting as Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
in Firefox - different messages, same symptom in other clients).
After some debugging, it appears all Jackson bytestream work eventually goes through BlockingBridge.withBlocking()
and always ends up with safeToRunInPlace
evaluating to true. In Blocking.rendezvous()
, smaller than 4kb payloads are on threads named DefaultDispatcher-worker-*
which can successfully park()
, where larger payloads are on threads named eventLoopGroupProxy-*-*
which have a Parking impl defined as ProhibitParking
. This is new behavior introduced in 1.4.2 as part of this commit: https://github.com/ktorio/ktor/commit/0dfada8cd8e438d724ce77e9cd0d8b09a6b8c6dc#diff-9004ca70aa325dbcfa7d3ce09d193306f9f33bfd9c7c7732875f008e294feb6d and throws the following:
java.lang.UnsupportedOperationException: Parking is prohibited on this thread. Most likely you are using blocking operation on the wrong thread/dispatcher that doesn't allow blocking. Consider wrapping you blocking code withContext(Dispatchers.IO) {...}.
This exception seems to lead to the coroutine ending up as a CompletedContinuation
, which is then cast to DispatchedContinuation
in CoroutineDispatcher.releaseInterceptedContinuation()
. This is what leads to the final representation of the stack trace that I see in logs:
Exception in thread "eventLoopGroupProxy-5-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@5f68303f, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@444cf2bb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:144)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:115)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.internal.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.internal.DispatchedContinuation are in unnamed module of loader 'app')
at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:104)
at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
... 23 more
A rather simple reproduction is available at https://github.com/rhurkes/ktor-playground/commit/989b512bd7209735fd4d223b839d20d307c533c3
NOTE: I did notice that stepping through code and flipping back and forth between payloads on either side of the 4kb line I could occasionally fail to reproduce this issue. There's definitely some magic in the dispatching that isn't 100% fully deterministic.
Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
failing on: val bookRequest = call.receive<Example>()
(full example):
package app.controllers
import app.models.Example
import app.services.ExampleService
import io.ktor.application.*
import io.ktor.auth.*
import io.ktor.auth.jwt.*
import io.ktor.features.*
import io.ktor.http.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import org.kodein.di.instance
class ExampleController(application: Application): Controller(application) {
private val exampleService: ExampleService by instance()
override fun Route.getRoutes() {
post("/") {
val bookRequest = call.receive<Example>()
exampleService.addExample(bookRequest)
call.respond(HttpStatusCode.Accepted)
}
}
}
Make Jackson serialization stream response
From user's PR:
The existing implementation of the JacksonConverter used a TextContent,
which meant that the ObjectMapper had to serialize the entire output
value to a String before responding.For large output objects, this causes significant allocation preassure.
It can avoid by instead streaming the output.
CORS denies simple request where the content type includes a character encoding
My understanding is that for a CORS simple request to run without prefight, part of the necessary criteria is that it can only have one of three ContentType
values: [application/x-www-form-urlencoded, multipart/form-data, text/plain]
.
Issue KTOR-358 correctly added a check to ensure this criteria is met. But the current implementation doesn't consider when character encoding is appended to ContentType. For example ContentType: text/plain;charset=UTF-8
. If the ContentType is specified in this manner, as is common/default on latest Chrome/Firefox, then the CORS feature will reject it, even though it is a simple request.
As far as I can tell from w3c there is no check required against the charset, rather the specification only cares about the mimeType. However, on reading mdn it says that the "only allowed values" are the three listed above which could be interpreted as not even charset is allowed.
As appending charset is the default on Chrome/Firefox, I think ktor should ignore parameters when parsing ContentType otherwise it will lead to devs setting allowNonSimpleContentTypes = true
which isn't great.
The current CORS.kt feature rejects these requests at line 102:
public suspend fun intercept(context: PipelineContext<Unit, ApplicationCall>) {
// ...
if (!allowNonSimpleContentTypes) {
val contentType = call.request.header(HttpHeaders.ContentType)?.let { ContentType.parse(it) }
if (contentType != null) {
if (contentType !in Configuration.CorsSimpleContentTypes) { // <--- issue
context.respondCorsFailed()
return
}
}
}
//...
}
Configuration.CorsSimpleContentTypes has no values inside parameters, but the parsed value for contentType will contain charset=UTF-8
. And so to fix this, then I think the check should be:
if(contentType.withoutParameters() !in Configuration.CorsSimpleContentTypes)
Thanks!
Autoreload embeddedServer configuration fails, if a module has default parameters
Ktor Version
1.4.2
Ktor Engine
Netty
JVM Version, Operating System and Relevant Context
JDK 11, Manjaro
Problem
- Project is generated by Ktor IDEA plugin, which if test modules are included, generates a module with default parameters.
- It fails with Module function provided as lambda cannot be unlinked for reload, if module has default parameters.
- Using method reference as in the example, does not make it work.
Example:
fun main(args: Array<String>): Unit {
embeddedServer(
Netty,
port = 8080,
host = "0.0.0.0",
watchPaths = listOf("some"),
module = Application::module,
).start(true)
}
fun Application.module(testing: Boolean = false): Unit {
routing {
get("/") {
call.respondText { "Hello" }
}
}
}
Running Result
Exception in thread "main" java.lang.RuntimeException: Module function provided as lambda cannot be unlinked for reload
at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.<init>(ApplicationEngineEnvironmentReloading.kt:58)
at io.ktor.server.engine.ApplicationEngineEnvironmentBuilder.build(ApplicationEngineEnvironment.kt:105)
at io.ktor.server.engine.ApplicationEngineEnvironmentKt.applicationEngineEnvironment(ApplicationEngineEnvironment.kt:44)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:55)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:36)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:34)
at some.ReloadKt.main(reload.kt:10)
It fails here, basically can't cast the default parameter method as KFunction<*>
public class ApplicationEngineEnvironmentReloading(
//...
) : ApplicationEngineEnvironment {
//...
private val moduleFunctionNames: List<String>? = run {
val configModules = config.propertyOrNull("ktor.application.modules")?.getList()
if (watchPatterns.isEmpty()) configModules
else {
val unlinkedModules = modules.map {
// Fails here
val fn = (it as? KFunction<*>)?.javaMethod
?: throw RuntimeException("Module function provided as lambda cannot be unlinked for reload")
val clazz = fn.declaringClass
val name = fn.name
"${clazz.name}.$name"
}
if (configModules == null)
unlinkedModules
else
configModules + unlinkedModules
}
//...
}
Workaround
fun Application.module() = module(false)
fun Application.module(testing: Boolean = false): Unit {...}
Test Infrastructure
Replace JUnit with kotlin.test
Replace JUnit with kotlin.test as much as possible
Reason
- Better Kotlin support, like assertEquals
- theoretical platform independent
Other
Create sample with native CIO client and `native-mt` coroutines
Remove copyTo usage from ServerPipeline
Handle failure in reading request body
@Test
fun testErrorInWritingPropagates() = testSuspend {
val client = HttpClient(factory)
val channel = ByteChannel(true)
channel.writeAvailable("text".toByteArray())
channel.close(IllegalStateException("Error on write"))
assertFailsWith<IllegalStateException>("Error on write") {
val result = client.post<String>("http://localhost:$serverPort/echo") {
body = channel
}
}
}
Apache: sends empty body without error
OkHttp: hangs
CIO: sends empty body without error
Android: succeed
Fix HTTP2 tests with new versions of java8
Blocking calls (ReentrantLock.lock) on Apache engine dispatcher threads
I'm using BlockHound to check if there are any blocking calls within my coroutines that aren't using IO dispatcher.
BlockHound reports the following occasion for Ktor client with Apache engine on thread ktor-apache-dispatcher-worker-4
.
Maybe it's a bug, maybe it's a misconfiguration of the dispatcher threads as CPU-bound.
reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:68)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:33)
at io.ktor.client.engine.apache.ApacheEngine.execute(ApacheEngine.kt:37)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Url(…) doesn't parse non-HTTP-like URL strings correctly
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1204
Url
is supposed to not just support HTTP-like URLs but any kind of URLs.
However the string parser seems to properly support only HTTP-like URLs:
Example
Url("mailto:marc@knaup.io").toString()
Url("file:///var/www").toString()
Expected
"mailto:marc@knaup.io"
"file:///var/www"
Actual
"mailto://localhost/marc@knaup.io"
"file://var/www"
Ktor 1.2.2
"response.readText()" never returns (probably infinite loop)
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/768
Ktor Version
1.0.0
Ktor Engine Used(client or server and name)
Apache
JVM Version, Operating System and Relevant Context
JDK 1.8.x, Win / Linux
Feedback
After about 1-2 hours of constant requests using ktor client it unexpectedly hangs on line:
val data = response.readText()
ktor will never pass this line. Even after 9 hours it's still on this line.
No exceptions are thrown and the process is not closed.
This might do something with the response from the server because it's unpredictable at all. It might happen after 10 minutes and might be after 6 hours.
Are there any suggestions what this can be or how can I investigate this issue?
Maybe, there are some timeouts that I need to set?
Thank you.
Ktor CIO client, connect timeout handling
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1720
- Using the following code (full example at https://github.com/rrva/ktor-client-connection-leak)
- Run the main method repeatedly. Sometimes, all futures created will not complete . Why? Expected behavior is that all created futures will complete eventually, so
Done with all
will be printed.
class Client(val fooServiceUrl: String) {
val client = HttpClient(CIO) {
engine {
requestTimeout = 500
endpoint.connectTimeout = 100
endpoint.connectRetryAttempts = 2
}
install(JsonFeature) {
serializer = KotlinxSerializer(
Json(
JsonConfiguration(
isLenient = true,
ignoreUnknownKeys = true,
serializeSpecialFloatingPointValues = true,
useArrayPolymorphism = true
)
)
)
}
}
suspend fun fetchFoo(ids: List<String>): List<Foo> {
return client.request {
header("User-Agent", "Foo")
parameter("input", ids.joinToString(","))
url("$fooServiceUrl/foo")
}
}
}
@Serializable
data class Foo(val id: String)
val client = Client("http://localhost:9090")
fun fetchFooAsync(ids:List<String>): CompletableFuture<List<Foo>> {
return CoroutineScope(Job() + Dispatchers.IO + MDCContext()).async {
try {
client.fetchFoo(ids)
} catch (e: Throwable) {
println(e.message)
listOf<Foo>(Foo("timeout"))
}
}.asCompletableFuture()
}
fun main() {
HttpServer.create(InetSocketAddress(9090), 1000).apply {
createContext("/foo") { http ->
http.responseHeaders.add("Content-type", "application/json")
http.sendResponseHeaders(200, 0)
PrintWriter(http.responseBody).use { out ->
out.println("""[{"id":"1"}]""")
}
}
start()
}
println("Start")
val requests = (1..100).map {
fetchFooAsync(listOf(it.toString()))
}
for (request in requests) {
println("Waiting for coroutine...")
println(request.get())
println("Done waiting")
}
println("Done with all")
}
Reproduced on 1.3.1
Extract flaky tests from `master` to the `flaky-tests` branch
We need this to improve general stability and measure how many flaky tests we have
Split ktor-server-test-host module
Ktor Client Null Pointer Exception (HTTPS)
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1890
Ktor Version and Engine Used (client or server and name)
1.3.1 (client)
Describe the bug
When sending a POST request over HTTPS, the ktor client will intermittently throw a NPE.
To Reproduce
Not sure exactly how to reproduce... given that the failure is intermittent. I get this NPE approximately 50% of the time.
Screenshots
kotlin.KotlinNullPointerException
at io.ktor.network.tls.TLSClientHandshake.handleServerDone(TLSClientHandshake.kt:327)
at io.ktor.network.tls.TLSClientHandshake.handleCertificatesAndKeys(TLSClientHandshake.kt:291)
at io.ktor.network.tls.TLSClientHandshake.negotiate(TLSClientHandshake.kt:155)
at io.ktor.network.tls.TLSClientHandshake$negotiate$1.invokeSuspend(TLSClientHandshake.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:561)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:727)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:667)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:655)
TestHttpClientEngine KNPE
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1891
Ktor Version and Engine Used (client or server and name)
1.3.2, test client/server, ktor-server-test-host
Describe the bug
Take this example.
fun main() {
withTestApplication(Application::helloModule) {
runBlocking {
val res = client.get<HttpResponse>("NonExistenceRoute")
println(res)
}
}
}
fun Application.helloModule() {
routing {
get {
call.respond(HttpStatusCode.OK, "Hello")
}
}
}
Throws KNPE because route doesn't exists. If you follow stack trace, it's obvious where KNPE was thrown.
It's pretty nice to have this kind of tests. Is TestHttpClientEngine
in experimental state? I also had another issue where the tests would stuck using multiparts.
Expected behavior
A response with 404 not found status.
Screenshots
Stack trace of the thrown exception,
Exception in thread "main" kotlin.KotlinNullPointerException
at io.ktor.server.testing.client.TestHttpClientEngine.execute(TestHttpClientEngine.kt:37)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:83)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Fix configuration if project without VPN and cache
Update kotlin to 1.4.20
NetworkOnMainThreadException when using a ktor mock server
Hello, we have got s strange issue which gives us a NetworkOnMainThreadException on our response when trying to parse.
Following situation:
- running a ktor server on localhost or different machine
- connecting via our kmp library on Android
- response parse -> NetworkOnMainThreadException
This happens only on Android. We recently fixed this problem by wrapping our calls into the IO dispatcher but it would be good to know what's the actual cause. This does not happen when we connect to the real backend. Also the connection via KotlinJs WebApp works fine. So this error doesn't make any sense to us.
How we response from ktor server:
val response = json.encodeToString(
PointOfSaleDto.serializer(),
mockPoiService.getPointOfSale()
)
call.respondText(response, ContentType.Application.Json)
How we read the response in our multiplatform lib (Android,iOS,Js):
val response = request<HttpResponse>(requestBuilder)
--------------------------------------
if (status.isSuccess()) {
return try {
ApiResult.Success(
status.value,
Json.decodeFromString(readText())
)
} catch (t: Throwable) {
....
}
}
Error: Could not find or load main class {MainClass}
ktor_version=1.4.2
embeddedServer Jetty: java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
I just can't explain why this is happening since my gradle dependencies definetly contain the missing class(es) (javax.servlet:javax.servlet-api:3.1.0).
The starting point of my application run is (run by Intellij IDEA):
@JvmStatic
fun main(args: Array<String>) {
start() // method contains the code below
}
when I try to run an embedded jetty server:
embeddedServer(Jetty, apiPort, module = serverMainModule) // AppServer.kt:312
this error occurs:
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at io.ktor.server.jetty.JettyApplicationEngineBase.<init>(JettyApplicationEngineBase.kt:42)
at io.ktor.server.jetty.JettyApplicationEngine.<init>(JettyApplicationEngine.kt:16)
at io.ktor.server.jetty.Jetty.create(Embedded.kt:14)
at io.ktor.server.jetty.Jetty.create(Embedded.kt:12)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:79)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:67)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:36)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:34)
at org.igmg.app.appserver.AppServer.start(AppServer.kt:312)
Not able to build iOS framework with Xcode 11.3.1 and ktor version 1.4.2 onwards. However, it works with ktor version 1.4.1
Attaching the error logs.
> Task :shared:linkDebugFrameworkIosArm64 FAILED
e: /Users/soumyajitdas/XCode Installer/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld invocation reported errors
The /Users/soumyajitdas/XCode Installer/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld command returned non-zero exit code: 1.
output:
Undefined symbols for architecture arm64:
"___darwin_check_fd_set_overflow", referenced from:
_io_ktor_network_interop_select_fd_add_wrapper1 in result.o
_io_ktor_network_interop_select_fd_isset_wrapper3 in result.o
ld: symbol(s) not found for architecture arm64
Execution failed for task ':shared:linkDebugFrameworkIosArm64'.
> Compilation finished with errors
1.4.1
released 24th September 2020
Client
[iOS] InvalidMutabilityException: mutation attempt of frozen
ONLY ON iOS, on android work correctly.
I've this class
open class BaseClient(val baseUrl: String) {
open val serializer = KotlinxSerializer(Json {
isLenient = true
ignoreUnknownKeys = true
})
open val logLevel = LogLevel.ALL
open val header = mapOf<String, String>()
open val client = HttpClient() {
install(JsonFeature) {
serializer = this@BaseClient.serializer
}
install(Logging) {
level = logLevel
}
defaultRequest {
header.forEach {
parameter(it.key, it.value)
}
if (this.method != HttpMethod.Get) contentType(ContentType.Application.Json)
}
}
}
On Android works correctly on iOS it crash with this error: InvalidMutabilityException: mutation attempt of frozen
If i delete all configuration in install command and configuration of default request, it doesn't crash on iOS.
So if I change from
install(JsonFeature) {
serializer = this@BaseClient.serializer
}
install(Logging) {
level = logLevel
}
defaultRequest {
header.forEach {
parameter(it.key, it.value)
}
if (this.method != HttpMethod.Get) contentType(ContentType.Application.Json)
}
to
install(JsonFeature)
install(Logging)
it works correctly.
Is this my bug or yours?
My Environment:
val commonMain by getting {
dependencies {
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.9-native-mt")
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.0.0")
implementation("io.ktor:ktor-client-serialization:1.4.1")
implementation("io.ktor:ktor-client-json:1.4.1")
implementation("io.ktor:ktor-client-serialization:1.4.1")
implementation("io.ktor:ktor-client-core:1.4.1")
implementation("io.ktor:ktor-client-cio:1.4.1")
implementation("io.ktor:ktor-client-logging:1.4.1")
}
}
val androidMain by getting {
dependencies {
implementation("io.ktor:ktor-client-android:1.4.1")
implementation("androidx.lifecycle:lifecycle-extensions:2.2.0")
implementation("androidx.lifecycle:lifecycle-viewmodel-ktx:2.2.0")
implementation("io.ktor:ktor-client-serialization-jvm:1.4.1")
}
}
val iosMain by getting {
dependencies {
implementation("io.ktor:ktor-client-ios:1.4.1")
implementation("io.ktor:ktor-client-serialization-iosx64:1.4.1")
implementation("com.github.florent37:multiplatform-livedata-ios:0.0.5")
}
}
InvalidMutabilityException: Configuration issues for ios
There is such a ktor configuration only for ios. Ignored ssl certificate.
actual fun get(): HttpClient {
return HttpClient(Ios) {
engine {
configureSession {
handleChallenge { session, task, challenge, completionHandler ->
val trust = challenge.protectionSpace.serverTrust
completionHandler(NSURLSessionAuthChallengeUseCredential, NSURLCredential.create(trust))
}
}
}
install(JsonFeature) {
serializer = KotlinxSerializer(
kotlinx.serialization.json.Json {
isLenient = true
ignoreUnknownKeys = true
}
)
}
install(Logging) {
logger = Logger.SIMPLE
//level = LogLevel.ALL
}
install(HttpTimeout) {
connectTimeoutMillis = TIMEOUT_MILLIS
requestTimeoutMillis = TIMEOUT_MILLIS
}
}
}
}```
I get an error on the platform: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.client.engine.ios.IosClientEngineConfig
Also tried this client configuration
```actual object HttpClientImpl {
actual fun get(): HttpClient {
return HttpClient {
engine {
Ios.config {
configureSession {
handleChallenge { session, task, challenge, completionHandler ->
val trust = challenge.protectionSpace.serverTrust
completionHandler(NSURLSessionAuthChallengeUseCredential, NSURLCredential.create(trust))
}
}
}
}
install(JsonFeature) {
serializer = KotlinxSerializer(
kotlinx.serialization.json.Json {
isLenient = true
ignoreUnknownKeys = true
}
)
}
install(Logging) {
logger = Logger.SIMPLE
//level = LogLevel.ALL
}
install(HttpTimeout) {
connectTimeoutMillis = TIMEOUT_MILLIS
requestTimeoutMillis = TIMEOUT_MILLIS
}
}
}
}```
Then the configuration block is ignored.
"InvalidMutabilityException: Frozen during lazy computation" when using by lazy for HttpClient
After updating PeopleInSpace (https://github.com/joreilly/PeopleInSpace) to following dependencies
const val kotlinCoroutines = "1.3.9-native-mt-2"
const val ktor = "1.4.1"
const val kotlinxSerialization = "1.0.0-RC2"
getting following crash in iOS client (as reported in https://github.com/joreilly/PeopleInSpace/issues/11)
Uncaught Kotlin exception: kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[MainDispatcher, Continuation @ $<init>$lambda-0COROUTINE$5]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
at 0 PeopleInSpaceSwiftUI 0x000000010e1e2b1c kfun:kotlin.Error#<init>(kotlin.String?;kotlin.Throwable?){} + 124
at 1 PeopleInSpaceSwiftUI 0x000000010e38b83c kfun:kotlinx.coroutines.CoroutinesInternalError#<init>(kotlin.String;kotlin.Throwable){} + 124
at 2 PeopleInSpaceSwiftUI 0x000000010e3d72e9 kfun:kotlinx.coroutines.DispatchedTask#handleFatalException(kotlin.Throwable?;kotlin.Throwable?){} + 953
at 3 PeopleInSpaceSwiftUI 0x000000010e3d6ec7 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3543
at 4 PeopleInSpaceSwiftUI 0x000000010e40a0b8 kfun:kotlinx.coroutines.DarwinMainDispatcher.dispatch$lambda-0#internal + 88
at 5 PeopleInSpaceSwiftUI 0x000000010e40a48e kfun:kotlinx.coroutines.DarwinMainDispatcher.$dispatch$lambda-0$FUNCTION_REFERENCE$43.invoke#internal + 62
at 6 PeopleInSpaceSwiftUI 0x000000010e40a4ee kfun:kotlinx.coroutines.DarwinMainDispatcher.$dispatch$lambda-0$FUNCTION_REFERENCE$43.$<bridge-UNN>invoke(){}#internal + 62
at 7 PeopleInSpaceSwiftUI 0x000000010e40b5a7 _6f72672e6a6574627261696e732e6b6f746c696e783a6b6f746c696e782d636f726f7574696e65732d636f7265_knbridge8 + 183
at 8 libdispatch.dylib 0x000000010f1c28ac _dispatch_call_block_and_release + 12
at 9 libdispatch.dylib 0x000000010f1c3a88 _dispatch_client_callout + 8
at 10 libdispatch.dylib 0x000000010f1d1f23 _dispatch_main_queue_callback_4CF + 1152
at 11 CoreFoundation 0x00007fff203a8276 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
at 12 CoreFoundation 0x00007fff203a2b06 __CFRunLoopRun + 2685
at 13 CoreFoundation 0x00007fff203a1b9e CFRunLoopRunSpecific + 567
at 14 GraphicsServices 0x00007fff2b773db3 GSEventRunModal + 139
at 15 UIKitCore 0x00007fff24660af3 -[UIApplication _run] + 912
at 16 UIKitCore 0x00007fff24665a04 UIApplicationMain + 101
at 17 PeopleInSpaceSwiftUI 0x000000010e1281cb main + 75 (/Users/mbonnin/git/PeopleInSpace/ios/PeopleInSpaceSwiftUI/PeopleInSpaceSwiftUI/AppDelegate.swift:5:7)
at 18 libdyld.dylib 0x00007fff20257415 start + 1
Caused by: kotlin.native.concurrent.InvalidMutabilityException: Frozen during lazy computation
at 0 PeopleInSpaceSwiftUI 0x000000010e1ea07d kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93
at 1 PeopleInSpaceSwiftUI 0x000000010e1e2c2b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91
at 2 PeopleInSpaceSwiftUI 0x000000010e1e2e7b kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91
at 3 PeopleInSpaceSwiftUI 0x000000010e218d6b kfun:kotlin.native.concurrent.InvalidMutabilityException#<init>(kotlin.String){} + 91
at 4 PeopleInSpaceSwiftUI 0x000000010e21b419 kfun:kotlin.native.concurrent.FreezeAwareLazyImpl.getOrInit#internal + 1641
at 5 PeopleInSpaceSwiftUI 0x000000010e21b95e kfun:kotlin.native.concurrent.FreezeAwareLazyImpl#<get-value>(){}1:0 + 670
at 6 PeopleInSpaceSwiftUI 0x000000010e13f183 kfun:com.surrus.common.remote.PeopleInSpaceApi.<get-client>#internal + 323
at 7 PeopleInSpaceSwiftUI 0x000000010e13feb7 kfun:com.surrus.common.remote.PeopleInSpaceApi.$fetchPeopleCOROUTINE$0#invokeSuspend(kotlin.Result<kotlin.Any?>){}kotlin.Any? + 2983
at 8 PeopleInSpaceSwiftUI 0x000000010e141a74 kfun:com.surrus.common.remote.PeopleInSpaceApi#fetchPeople(){}com.surrus.common.remote.AstroResult + 244
at 9 PeopleInSpaceSwiftUI 0x000000010e146666 kfun:com.surrus.common.repository.PeopleInSpaceRepository.$fetchAndStorePeopleCOROUTINE$2.invokeSuspend#internal + 886
at 10 PeopleInSpaceSwiftUI 0x000000010e146bb4 kfun:com.surrus.common.repository.PeopleInSpaceRepository.fetchAndStorePeople#internal + 244
at 11 PeopleInSpaceSwiftUI 0x000000010e1487da kfun:com.surrus.common.repository.PeopleInSpaceRepository.$<init>$lambda-0COROUTINE$5.invokeSuspend#internal + 602
at 12 PeopleInSpaceSwiftUI 0x000000010e20bba6 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 758
at 13 PeopleInSpaceSwiftUI 0x000000010e3d6be2 kfun:kotlinx.coroutines.DispatchedTask#run(){} + 2802
at 14 PeopleInSpaceSwiftUI 0x000000010e40a0b8 kfun:kotlinx.coroutines.DarwinMainDispatcher.dispatch$lambda-0#internal + 88
... and 14 more common stack frames skipped
CoreSimulator 732.17 - Device: iPhone 11 (8BACA2EC-79DF-49D0-937D-AFA747B52692) - Runtime: iOS 14.0 (18A372) - DeviceType: iPhone 11
(lldb)
The particular code that's triggering this is in PeopleInSpaceRepository
. That uses Ktor to make api request and then stores in db (but see same issue if I only do ktor request there)
init {
GlobalScope.launch(Dispatchers.Main) {
fetchAndStorePeople()
}
}
Ktor OkHttp client uses blocking calls inside Coroutine Dispatcher (ktor-okhttp-dispatcher) with default maxPoolSize = 4.
Ktor OkHttp client uses blocking calls inside Coroutine Dispatcher (ktor-okhttp-dispatcher) with default maxPoolSize = 4.
I guess the default HttpClientEngineConfig.threadsCount for OkHttp should be increased.
java.net.SocketInputStream.socketRead0(SocketInputStream.java)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at okio.InputStreamSource.read(JvmOkio.kt:90)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
at okio.RealBufferedSource.read(RealBufferedSource.kt:188)
at okhttp3.internal.http1.Http1ExchangeCodec$AbstractSource.read(Http1ExchangeCodec.kt:331)
at okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.kt:412)
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:276)
at okio.RealBufferedSource.read(RealBufferedSource.kt:82)
at io.ktor.client.engine.okhttp.OkHttpEngineKt$toChannel$1$invokeSuspend$$inlined$use$lambda$1.invoke(OkHttpEngine.kt:154)
at io.ktor.client.engine.okhttp.OkHttpEngineKt$toChannel$1$invokeSuspend$$inlined$use$lambda$1.invoke(OkHttpEngine.kt)
at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1593)
at io.ktor.utils.io.ByteBufferChannel.write$suspendImpl(ByteBufferChannel.kt:1620)
at io.ktor.utils.io.ByteBufferChannel.write(ByteBufferChannel.kt)
at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.write$default(ByteWriteChannel.kt:100)
at io.ktor.client.engine.okhttp.OkHttpEngineKt$toChannel$1.invokeSuspend(OkHttpEngine.kt:152)
at io.ktor.client.engine.okhttp.OkHttpEngineKt$toChannel$1.invoke(OkHttpEngine.kt)
at io.ktor.utils.io.CoroutinesKt$launchChannel$job$1.invokeSuspend(Coroutines.kt:129)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
OkHTTP client engine tries to close the connection twice during the closing handshake
Here's how I noticed:
When I cancel the coroutine containing the WebSocket connection, KTor closes the connection with a 1011
. If the server then responds by closing the connection without a status code, KTor will get a 1005
and it'll try to pass it to OkHTTP.
However, 1005
is a reserved status, so OkHTTP refuses it when Ktor passes it in the following lines: https://github.com/ktorio/ktor/blob/6053fa44a61fdfc5474ff909c0224e56877b5da1/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L62
Here's an example of a traceback you'd get:
java.util.concurrent.CancellationException: ActorCoroutine was cancelled
at kotlinx.coroutines.ExceptionsKt.CancellationException(Exceptions.kt:22)
at kotlinx.coroutines.channels.ActorCoroutine.onCancelling(Actor.kt:134)
at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:916)
at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:875)
at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:840)
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.IllegalArgumentException: Code 1005 is reserved and may not be used.
at okhttp3.internal.ws.WebSocketProtocol.validateCloseCode(WebSocketProtocol.kt:134)
at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:435)
at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:427)
at io.ktor.client.engine.okhttp.OkHttpWebsocketSession$outgoing$1.invokeSuspend(OkHttpWebsocketSession.kt:62)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
I think there are technically two different issues happening at the same time:
- Ktor is trying to close the connection twice. The first time with a
1011
status (which I can confirm in the server logs) and the second time with a1005
(because the server didn't provide a closing status code). - Ktor is trying to close the connection with a reserved status code.
"Unfinished workers detected" using client on native
To reproduce run ./gradlew runDebugExecutableMacosX64
on the attached project.
As a result, I get unexpected Unfinished workers detected, 1 workers leaked!
warning and the program finishes with 134 exit code.
As a workaround, I can add Platform.isMemoryLeakCheckerActive = false
line to prevent this behavior.
Here is a code example that after execution reports Unfinished workers detected, 100 workers leaked!
warning:
fun main() {
runBlocking {
repeat(100) {
val client = HttpClient(Curl)
val data = client.get<String> {
url("http://google.com")
}
println(data)
client.close()
}
}
}
ResponseException is no longer serializable starting from 1.4.0 (breaking change)
Previously a ResponseException could be serialized. See: https://github.com/ktorio/ktor/issues/1256. The Transient annotation was removed in a recent commit for the HttpResponse
field. Now an exception is thrown when serializing a ResponseException.
Stack trace:
io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
java.io.NotSerializableException: io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
at ExceptionsTest.serialize(ExceptionsTest.kt:26)
at ExceptionsTest.testResponseExceptionSerializable(ExceptionsTest.kt:17)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Here is a test to reproduce the issue:
import io.ktor.client.call.*
import io.ktor.client.features.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.util.date.*
import io.ktor.utils.io.*
import java.io.*
import kotlin.coroutines.*
import kotlin.test.*
class ExceptionsTest {
@Test
fun testResponseExceptionSerializable() {
val exception = createResponseException()
val serialized = serialize(exception)
val deserialized = deserialize(serialized)
deserialized as ResponseException
}
private fun serialize(obj: Any): ByteArray {
val b = ByteArrayOutputStream()
val o = ObjectOutputStream(b)
o.writeObject(obj)
return b.toByteArray()
}
private fun deserialize(bytes: ByteArray): Any? {
val b = ByteArrayInputStream(bytes)
val o = ObjectInputStream(b)
return o.readObject()
}
}
private fun createResponseException(): ResponseException = ResponseException(object : HttpResponse() {
override val call: HttpClientCall
get() = TODO("Not yet implemented")
override val status: HttpStatusCode
get() = TODO("Not yet implemented")
override val version: HttpProtocolVersion
get() = TODO("Not yet implemented")
override val requestTime: GMTDate
get() = TODO("Not yet implemented")
override val responseTime: GMTDate
get() = TODO("Not yet implemented")
override val content: ByteReadChannel
get() = TODO("Not yet implemented")
override val headers: Headers
get() = TODO("Not yet implemented")
override val coroutineContext: CoroutineContext
get() = TODO("Not yet implemented")
override fun toString(): String = "FakeCall"
}, cachedResponseText = "Fake text")
Make curl engine option sslVerify public
Provide/publish internal option to skip TLS verification for curl client engine
Client: NPE in FormDataContentKt -> Input.copyTo
I've received the following exception report for an Android app.
It was sporadic and I cannot reproduce the problem. The problem seems to lie within Ktor itself so I'll simply share what I can.
FATAL EXCEPTION: k.a0 Dispatcher = okhttp3.OkHttpClient Dispatcher
java.lang.NullPointerException
at h.b.a.g.j.b$a.l(Unknown Source:58) = io.ktor.client.request.forms.FormDataContentKt$copyTo$2.invokeSuspend(java.lang.Object)
at kotlin.c0.k.a.a.n(Unknown Source:9) = kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(java.lang.Object)
at kotlinx.coroutines.a1.run(Unknown Source:93)
at kotlinx.coroutines.d3.b.b0(Unknown Source:0)
at kotlinx.coroutines.d3.b$a.c(Unknown Source:14)
at kotlinx.coroutines.d3.b$a.m(Unknown Source:28)
at kotlinx.coroutines.d3.b$a.run(Unknown Source:0)
This is the relevant mapping:
io.ktor.client.request.forms.FormDataContentKt$copyTo$2 -> h.b.a.g.j.b$a:
io.ktor.utils.io.WriterSuspendSession p$ -> p
java.lang.Object L$0 -> q
io.ktor.utils.io.core.Input $this_copyTo -> s
int label -> r
java.lang.Object invoke(java.lang.Object,java.lang.Object) -> M
kotlin.coroutines.Continuation create(java.lang.Object,kotlin.coroutines.Continuation) -> d
java.lang.Object invokeSuspend(java.lang.Object) -> l
The relevant code from Ktor:
https://github.com/ktorio/ktor/blob/a6d68420ee0b272cd09d235057427a6601736487/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/FormDataContent.kt#L154
private suspend fun Input.copyTo(channel: ByteWriteChannel) {
if (this is ByteReadPacket) {
channel.writePacket(this)
return
}
channel.writeSuspendSession {
while (!this@copyTo.endOfInput) {
tryAwait(1)
val buffer = request(1)!! // <-- likely cause
val size = this@copyTo.readAvailable(buffer)
if (size < 0) continue
written(size)
}
}
}
CIO: client engine exceptions are both logged and thrown
In the CIO client engine exceptions like SocketTimeoutException
are logged to standard error by the coroutines' uncaught exception handler and thrown by client.get()
.
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.*
suspend fun main(): Unit = withContext(Dispatchers.Default) {
HttpClient(CIO) {
install(HttpTimeout) {
socketTimeoutMillis = 5
}
}.use { client ->
try {
client.get<HttpResponse>("http://www.google.com").readText()
} catch (e: Throwable) {
println("Caught: $e")
}
}
}
Output:
Exception in thread "kotlinx.coroutines.DefaultExecutor" java.net.SocketTimeoutException
at io.ktor.network.util.UtilsKt$withSocketTimeout$2.invokeSuspend(Utils.kt:23)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32)
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:188)
at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:122)
at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:111)
at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)
at kotlinx.coroutines.CancellableContinuationImpl.cancel(CancellableContinuationImpl.kt:177)
at kotlinx.coroutines.CancellableContinuationImpl.parentCancelled$kotlinx_coroutines_core(CancellableContinuationImpl.kt:184)
at kotlinx.coroutines.ChildContinuation.invoke(JobSupport.kt:1484)
at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:1511)
at kotlinx.coroutines.JobSupport.tryMakeCancelling(JobSupport.kt:792)
at kotlinx.coroutines.JobSupport.makeCancelling(JobSupport.kt:752)
at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:668)
at kotlinx.coroutines.JobSupport.cancelCoroutine(JobSupport.kt:655)
at kotlinx.coroutines.TimeoutCoroutine.run(Timeout.kt:128)
at kotlinx.coroutines.EventLoopImplBase$DelayedRunnableTask.run(EventLoop.common.kt:497)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:68)
at java.base/java.lang.Thread.run(Thread.java:832)
Caught: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=http://www.google.com/, socket_timeout=5] ms
HttpRedirect feature alters Location header value
Currently HttpRedirect
feature alters Location
header value by escaping query parameters, but it's significant in case of signed URLs verifications.
val httpClient = HttpClient(Apache) {
followRedirects = true
}
val response = httpClient.get<HttpResponse>("https://dl.bintray.com/jetbrains/markdown/org/jetbrains/markdown-jvm/0.2.0.pre-61/markdown-jvm-0.2.0.pre-61.jar")
assertTrue(response.status.isSuccess())
The original response contains Location
like that:
https://akamai.bintray.com/22/225b067044aa56f36590ef56d41e256cd1d0887b176bfdeec123ecccc6057790?__gda__=exp=1604350711~hmac=417cbd5a97b4c499e2cf7e9eae5dfb9ad95b42cb3ff76c5fb0fae70e2a42db9c&...
But Ktor executes URL encode for query parameters and uses the following value:
https://akamai.bintray.com/22/225b067044aa56f36590ef56d41e256cd1d0887b176bfdeec123ecccc6057790?__gda__=exp%3D1604350711~hmac%3D417cbd5a97b4c499e2cf7e9eae5dfb9ad95b42cb3ff76c5fb0fae70e2a42db9c&...
CIO Engine's HttpClient may fail when trying to send large size binary data.
Ktor Version and Engine Used (client or server and name)
Client CIO, 1.4.1
Describe the bug
HttpClient with CIO Engine sometimes sends an invalid multipart/form-data request when it includes large size binary data.
When it comes to invalid requests, it does not seem to contain a trailing multipart/form-data request boundary.
(There doesn't seem to be enough content in the request body.)
java.nio.BufferOverflowException
seems to occur in the following process.
https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/FormDataContent.kt#L123
This behavior is not reproduced in the case of HttpClient with Apache Engine.
To Reproduce
Sample code is below.
@KtorExperimentalAPI
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
routing {
post("/multipart") {
val multipartData = call.receiveMultipart()
multipartData.forEachPart {
if (it is PartData.FileItem) {
println(
withContext(Dispatchers.IO) {
it.streamProvider().readAllBytes().size
}
)
}
}
call.respondText("test")
}
get("/execute/cio") {
repeat(10) {
request(HttpClient(CIO))
}
call.respondText("success")
}
get("/execute/apache") {
repeat(10) {
request(HttpClient(Apache))
}
call.respondText("success")
}
}
}
suspend fun request(client: HttpClient) {
val body = formData {
val headers = headersOf(
HttpHeaders.ContentDisposition,
ContentDisposition.File
.withParameter(ContentDisposition.Parameters.Name, "file")
.withParameter(ContentDisposition.Parameters.FileName, "test.jpg")
.toString()
)
appendInput("file", headers) {
// dummy 10MB binary file
ByteArray(10_000_000).inputStream().asInput()
}
}
client.use {
it.submitFormWithBinaryData<Unit>(
url = "http://localhost:8080/multipart",
formData = body
)
}
}
(full code is in https://github.com/ma2Cta/ktor-multipart-request-test/blob/main/src/Application.kt )
-
Start the server.
-
Run
$ curl http://localhost:8080/execute/apache
a few times.
10 times 10000000
will be printed as expected.
- Run
$ curl http://localhost:8080/execute/cio
a few times.
You won't get 10000000
10 times, and you'll probably get a number less than 10000000
the first few times
Expected
The correct request is sent even in the case of CIO Engine.
Timeout feature: android engine throws Java's SocketTimeoutException instead of ConnectTimeoutException
While using Ktor client, with timeout feature, the following instruction can throw a java.net.SocketTimeoutException
instead of the expected ConnectTimeoutException
. The stack trace is in the attachments.
PS: The exceptions section in the Timeout feature documentation is outdated.
Apache client engine sometimes hits an unrecoverable socket timeout when using ChannelWriterContent
Test to reproduce below. Tested on two different machines (Windows and macOS), works fine with CIO
. With 500 test iterations, happens at least once most times.
import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.apache.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.http.*
import io.ktor.http.content.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.cio.*
import io.ktor.server.engine.*
import io.ktor.utils.io.*
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.RepeatedTest
import java.util.concurrent.TimeUnit
class KtorApacheBugTest {
private val port = 8000
private val server = embeddedServer(
factory = CIO,
port = port,
) {
routing {
put("/") {
call.receiveChannel().discard()
call.respond(HttpStatusCode.OK)
}
}
}
private val client = HttpClient(Apache) {
install(HttpTimeout) {
socketTimeoutMillis = 10_000
connectTimeoutMillis = 10_000
}
}
@BeforeEach
fun setUp() {
server.start()
}
@AfterEach
fun tearDown() {
client.close()
server.stop(0, 1, TimeUnit.SECONDS)
}
@RepeatedTest(500)
fun `eventually hits a socket timeout`(): Unit = runBlocking {
val chunk = (0 until 10_000).map { it.toByte() }.toByteArray()
client.put<String>("http://localhost:${port}") {
body = ChannelWriterContent(
body = {
writeFully(chunk)
},
contentType = null,
)
}
}
}
Another simple test (seems the issue is with ChannelWriterContent
) below.
This works:
@RepeatedTest(500)
fun `does not hit a socket timeout`(): Unit = runBlocking {
client.put<String>("http://localhost:${port}") {
body = LocalFileContent(File("gradle/wrapper/gradle-wrapper.jar"))
}
}
This does not:
@RepeatedTest(500)
fun `eventually hits a socket timeout`(): Unit = runBlocking {
val chunk = File("gradle/wrapper/gradle-wrapper.jar").readBytes()
client.put<String>("http://localhost:${port}") {
body = ChannelWriterContent(
body = {
writeFully(chunk)
},
contentType = null,
)
}
}
Sending the same via ByteArrayContent
works fine as well.
Ktor 1.4.1 fails to upload file (submit form) when logging level = LogLevel.BODY
I am using ktor 1.4.1 in my multiplatform project to submit a file.
the weird thing is that whenever I change the log level to BODY the form parameters are corrupt, but when it is INFO it works correctly.
install(Logging) {
logger = object : Logger {
override fun log(message: String) {
println(message)
}
}
level = LogLevel.BODY // FAILS to submit
level = LogLevel.INFO // OK
}
here is the code to submit file
httpClient.submitFormWithBinaryData(
"https://myendpoint",
formData {
appendInput(
key = "file",
headers = Headers.build {
append(ContentDisposition, " filename=video.mp4")
}
) { buildPacket { writeFully(data) } }
}
)
And when I tried to debug, I found out that when level = LogLevel.BODY
ktor does not send
Content-Type: multipart/form-data; boundary=XXX
in request. It is only available in logs but not actually sent. Maybe logger consumes it or sth like this happens.
Overload resolution ambiguity: HttpClient.ws
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.features.websocket.*
suspend fun main() {
HttpClient(CIO).ws("http://localhost") {}
}
Overload resolution ambiguity:
public suspend fun HttpClient.ws(urlString: String, request: HttpRequestBuilder.() -> Unit = ..., block: suspend DefaultClientWebSocketSession.() -> Unit): Unit defined in io.ktor.client.features.websocket
public suspend fun HttpClient.ws(urlString: String, request: HttpRequestBuilder.() -> Unit = ..., block: suspend DefaultClientWebSocketSession.() -> Unit): Unit defined in io.ktor.client.features.websocket
There's one in core-jvm
and one in cio-jvm
.
Content-type header is missing in Request when using LogLevel.BODY
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1690
Ktor Version and Engine Used (client or server and name)
version: 1.3.1
lib: ktor-client-cio
additional modules: ktor-client-logging-jvm
Describe the bug
Content-type header is missing when LogLevel.BODY
is set. (also I am using multi-part body, not sure if relevant). Header exists if removing logging or setting LogLevel.HEADER
.
To Reproduce
Steps to reproduce the behavior:
- Set logging
HttpClient {
install(Logging) {
logger = Logger.ANDROID
level = LogLevel.BODY
}
}
- Do multi-form request
val response = httpClient.post<String> {
url("http://someurl")
body = MultiPartFormDataContent(formData {
append(
"file",
"file.txt",
ContentType.parse("text/plain"),
contentBytes.size.toLong()
) {
writeFully(contentBytes)
}
})
- Observe missing
Content-Type: multipart/form-data; <...>
header
Expected behavior
Content-Type: multipart/form-data; <...>
in request exists
Jetty client engine with high concurrency randomly freezes, throws NPEs, ISEs or EofExceptions
Exception in thread "main" java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:222)
at org.eclipse.jetty.io.SelectorManager.connect(SelectorManager.java:193)
at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:409)
at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:382)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$connect$2.invoke(JettyHttpRequest.kt:61)
at io.ktor.client.engine.jetty.JettyHttpRequestKt$connect$2.invoke(JettyHttpRequest.kt)
at io.ktor.client.engine.jetty.UtilsKt.withPromise(utils.kt:14)
at io.ktor.client.engine.jetty.JettyHttpRequestKt.connect(JettyHttpRequest.kt:59)
at io.ktor.client.engine.jetty.JettyHttpRequestKt.executeRequest(JettyHttpRequest.kt:31)
at io.ktor.client.engine.jetty.JettyHttp2Engine.execute(JettyHttp2Engine.kt:38)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Suppressed: java.lang.NullPointerException
... 17 more
Suppressed: java.lang.NullPointerException
... 17 more
Suppressed: java.lang.NullPointerException
... 17 more
Suppressed: java.lang.NullPointerException
... 17 more
import io.ktor.client.*
import io.ktor.client.engine.jetty.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.*
suspend fun main() = withContext(Dispatchers.Default) {
HttpClient(Jetty) {
expectSuccess = false
install(HttpTimeout) {
connectTimeoutMillis = 2_000
socketTimeoutMillis = 2_000
}
engine {
threadsCount = 200
}
}.use { client ->
coroutineScope {
repeat(5) {
repeat(200) {
launch {
client.get<HttpResponse>("https://esi.evetech.net/latest/markets/10000040/orders/?page=1")
.readText()
.let { println(it.length) }
}
}
}
}
}
println()
println("Done")
}
Native: ResponseObserver hangs coroutine indefinitely
Here's my reproducer:
launch {
try {
val client = HttpClient()
ResponseObserver.install(ResponseObserver {
// Whatever
println("Bonjour!")
}, client)
client.get<HttpResponse>("https://www.google.com/")
onSuccess()
} catch (t: Throwable) {
t.printStackTrace()
onError()
}
}
Reproducer project here: https://github.com/Ribesg/kotlin-reproducer/tree/ktor-1095
Just clone, open in XCode 11.x and Run (tested on simulator).
CertificatePinner buildErrorMessage outputs CertificatesInfo reference and variable name (HASH_ALGORITHM_SHA_256) instead of value
When making the certificate pinner fail on iOS, it outputs this:
Certificate pinning failure!
Peer certificate chain:
io.ktor.client.engine.ios.certificates.CertificatesInfo@3d83458.HASH_ALGORITHM_SHA_256fEbY8nKEFY2ODyxn2P0kyIq+C2/Rkke8ifByvsmamLw=: *.domain.com
io.ktor.client.engine.ios.certificates.CertificatesInfo@3d83458.HASH_ALGORITHM_SHA_256hETpgVvaLC0bvcGG3t0cuqiHvr4XyP2MTwCiqhgRWwU=: GlobalSign RSA OV SSL CA 2018
io.ktor.client.engine.ios.certificates.CertificatesInfo@3d83458.HASH_ALGORITHM_SHA_256cGuxAXyFXFkWm61cF4HPWX8S0srS9j0aSqN0k4AP+4A=: GlobalSign
It inserts the reference to the CertificatesInfo object followed by the name of the variable HASH_ALGORITHM_SHA_256.
Expected output can be read in the documentation of CertificatePinner.
This is a bug in the error message building in CertificatePinner.buildErrorMessage().
CIO: client throws ConnectTimeoutException when endpoint.connectRetryAttempts = 0
The following code fails with a ConnectTimeoutException
.
Maybe the property connectRetryAttempts
is mislabeled and actually means the total number of attempts, not the number of retry attempts.
suspend fun main(): Unit = withContext(Dispatchers.Default) {
HttpClient(CIO) {
engine {
endpoint.connectRetryAttempts = 0
}
}.use { client ->
client.get<String>("https://www.google.com")
}
}
Exception in thread "main" io.ktor.network.sockets.ConnectTimeoutException: Connect timeout has been expired [url=https://www.google.com/, connect_timeout=unknown ms]
at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException(HttpTimeout.kt:164)
at io.ktor.client.features.HttpTimeoutKt.ConnectTimeoutException$default(HttpTimeout.kt:163)
at io.ktor.client.engine.cio.Endpoint.getTimeoutException(Endpoint.kt:208)
at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:200)
at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Client: InvalidCacheStateException when Vary header differs for 200 and 304 responses
I perform a lot of requests in parallel with relatively short-lived Expires values. That leads to plenty of exceptions like this:
Exception in thread "main" io.ktor.client.features.cache.InvalidCacheStateException: The entry for url: https://esi.evetech.net/latest/markets/10000035/orders/?page=1 was removed from cache
at io.ktor.client.features.cache.HttpCache$Companion$install$2.invokeSuspend(HttpCache.kt:104)
at io.ktor.client.features.cache.HttpCache$Companion$install$2.invoke(HttpCache.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
at io.ktor.client.HttpClient$1.invokeSuspend(HttpClient.kt:141)
at io.ktor.client.HttpClient$1.invoke(HttpClient.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:69)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
"FreezingException: freezing of InvokeOnCompletion has failed" using native-mt coroutines
Hi.
I have a problem on Ktor client native
.
I’ve been struggling to make the code below works after migrating kotlin to 1.4.0
version
while migrating, I changed coroutine version to 1.3.9-native-mt
class TestFreezingViewModel(val client: HttpClient) {
fun call() {
CoroutineScope(Dispatchers.Main).launch(start = CoroutineStart.LAZY) {
try {
client.get<String>("https://test.com") {}
} catch (e: Throwable) {
e.printStackTrace()
}
}.start()
}
}
swift side code
TestFreezingViewModel().call()
while the code running, The error below occurs
kotlin.native.concurrent.FreezingException: freezing of InvokeOnCompletion[InvokeOnCompletion@1b76e08] has failed, first blocker is HttpClient[io.ktor.client.engine.ios.IosClientEngine@1b1b748]
at 0 kotlinIOS 0x000000010bf93fed kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93
at 1 kotlinIOS 0x000000010bf8cb9b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91
at 2 kotlinIOS 0x000000010bf8cdeb kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91
at 3 kotlinIOS 0x000000010bfc2c13 kfun:kotlin.native.concurrent.FreezingException#<init>(kotlin.Any;kotlin.Any){} + 643
at 4 kotlinIOS 0x000000010bfc4297 ThrowFreezingException + 231
at 5 kotlinIOS 0x000000010c0b2f4f FreezeSubgraph + 2815
at 6 kotlinIOS 0x000000010c0cc17b Kotlin_Worker_freezeInternal + 27
at 7 kotlinIOS 0x000000010bfc2d3e kfun:kotlin.native.concurrent#freeze@0:0(){0§<kotlin.Any?>}0:0 + 62
at 8 kotlinIOS 0x000000010c10c749 kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Boolean;kotlin.Boolean;kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 1881
at 9 kotlinIOS 0x000000010c10bf76 kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 198
at 10 kotlinIOS 0x000000010c33b15b kfun:io.ktor.client.engine#createCallContext@io.ktor.client.engine.HttpClientEngine(kotlinx.coroutines.Job){}kotlin.coroutines.CoroutineContext + 1451
at 11 kotlinIOS 0x000000010c302d08 kfun:io.ktor.client.engine.HttpClientEngine.$executeWithinCallContextCOROUTINE$9.invokeSuspend#internal + 1048
at 12 kotlinIOS 0x000000010c303444 kfun:io.ktor.client.engine.HttpClientEngine.executeWithinCallContext#internal + 308
at 13 kotlinIOS 0x000000010c30420a kfun:io.ktor.client.engine.HttpClientEngine.$install$lambda-0COROUTINE$10.invokeSuspend#internal + 1562
at 14 kotlinIOS 0x000000010c304bf8 kfun:io.ktor.client.engine.HttpClientEngine.$install$lambda-0COROUTINE$10.invoke#internal + 312
at 15 kotlinIOS 0x000000010c29da5f kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1135
at 16 kotlinIOS 0x000000010c29d0ab kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 395
at 17 kotlinIOS 0x000000010c29d50a kfun:io.ktor.util.pipeline.SuspendFunctionGun.execute#internal + 474
at 18 kotlinIOS 0x000000010c295759 kfun:io.ktor.util.pipeline.Pipeline#execute(1:1;1:0){}1:0 + 361
at 19 kotlinIOS 0x000000010c326be5 kfun:io.ktor.client.features.HttpSend.DefaultSender.$executeCOROUTINE$27.invokeSuspend#internal + 1365
at 20 kotlinIOS 0x000000010c327244 kfun:io.ktor.client.features.HttpSend.DefaultSender.execute#internal + 308
at 21 kotlinIOS 0x000000010c324fbc kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0COROUTINE$26.invokeSuspend#internal + 2236
at 22 kotlinIOS 0x000000010c3260f8 kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0COROUTINE$26.invoke#internal + 312
at 23 kotlinIOS 0x000000010c29da5f kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1135
at 24 kotlinIOS 0x000000010c29d0ab kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 395
at 25 kotlinIOS 0x000000010c29d2a5 kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceedWith#internal + 213
at 26 kotlinIOS 0x000000010c3173eb kfun:io.ktor.client.features.HttpCallValidator.Companion.$install$lambda-0COROUTINE$17.invokeSuspend#internal + 907
at 27 kotlinIOS 0x000000010c317c58 kfun:io.ktor.client.features.HttpCallValidator.Companion.$install$lambda-0COROUTINE$17.invoke#internal + 312
at 28 kotlinIOS 0x000000010c29da5f kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1135
at 29 kotlinIOS 0x000000010c29d0ab kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 395
when Coroutine’s invokeOnCompletion
is invoked, freeze()
is called.
and it tells that HttpClient
can’t be frozen
When I checked the source code
HttpClient.init
calls preventFreeze()
I would like to know the intention of preventFreeze
and what is the proper approach to use on developer’s side.
please check if this is bug, other wise, kindly let me share some advice.
Thanks
CIO: java.io.EOFException when server requires client certificate and restricts the authority
CIO tries to read more bytes as available in the packet, see pull request https://github.com/ktorio/ktor/pull/2063.
This results into a crash, and no retry is possible, because false handling of the authoritiesSize.
java.io.EOFException: Premature end of stream: expected 2 bytes
Example:
reading | remainingBefore | remainingAfter | content | position | |
---|---|---|---|---|---|
before | 281 | 281 | |||
reading authorizationSize | 2 | 281 | 279 | 279 | |
reading size | 2 | 279 | 277 | 91 | |
reading CA 1 | 91 | 277 | 186 | CA... | 91 (should 93) |
reading size | 2 | 186 | 184 | 91 | |
reading CA 2 | 91 | 184 | 93 | CA | 182 (should 186) |
reading size | 2 | 93 | 91 | 91 | |
reading CA 3 | 91 | 91 | 0 | CA | 273 (should 279) |
Calling HttpStatement#toString more than once throws IllegalArgumentException
Ktor v1.4.0
This can be trivially reproduced with the following code:
fun main() {
// Using OkHttp for illustrative purposes; it doesn't matter what engine
val client = HttpClient(OkHttp)
val statement = HttpStatement(HttpRequestBuilder(), client)
println(statement.toString())
println(statement.toString())
}
Expected output:
HttpStatement[http://localhost/]
HttpStatement[http://localhost/]
Actual output:
HttpStatement[http://localhost/]
Exception in thread "main" java.lang.IllegalArgumentException: ParametersBuilder can only build a single Parameters instance
at io.ktor.http.ParametersBuilder.build(Parameters.kt:32)
at io.ktor.http.URLBuilder.appendTo(URLBuilder.kt:67)
at io.ktor.http.URLBuilder.buildString(URLBuilder.kt:81)
at io.ktor.client.statement.HttpStatement.toString(HttpStatement.kt:138)
at MainKt.main(Main.kt:11)
at MainKt.main(Main.kt)
Process finished with exit code 1
This is because of:
HttpStatement#toString
callsURLBuilder#buildString
, which callsURLBuilder#appendTo
URLBuilder#appendTo
callsParametersBuilder#build
ParametersBuilder#build
can only be called once, per this assertion:
require(!built) { "ParametersBuilder can only build a single Parameters instance" }
Implement runtime check of using `native-mt` coroutines
Ktor websocket client passes configured max frame as timeout millis
The client passes maxFrameSize instead of timeoutMillis as the third argument.
1.4.0: breaking change by making response nullable in ResponseException
Link to breaking change: https://github.com/ktorio/ktor/commit/9785a7a82e89881ad279a77ac6adedfe621fb0b8#r41593709
Either this change should have been kept for a future version, or this version should be labeled 2.0.0
Parser Exception in header with character code 1 not allowed
The following URL
https://data.smartdublin.ie/cgi-bin/rtpi
returns valid results with other HTTP clients, including IntelliJ HTTP Requests. However with Ktor, independently of engine, it throws a parser exception (side note, it would be good to indicate the header in the exception result)
Exception in thread "main" io.ktor.http.cio.ParserException: Character with code 1 is not allowed in header names,
HTTP/1.1 200 OKServer: nginx/1.17.8Date: Sun, 19 Jul 2020 07:12:42 GMTContent-Type: application/json; charset=utf-8Content-Length: 1596054Connection: keep-aliveX-Powered-By: ASP.NETSet-Cookie: visid_incap_1822411=6ycr4UzsSv+6sdm+QnNmpmnyE18AAAAAQUIPAAAAAACm9JYEnYu4a0fTA7VxKyxu; expires=Sun, 18 Jul 2021 08:40:35 GMT; HttpOnly; path=/; Domain=.nationaltransport.ieSet-Cookie: incap_ses_535_1822411=vjDoPwoqvTdguklauLNsB2nyE18AAAAAoieq1zxaD46bWhzEJyQd3w==; path=/; Domain=.nationaltransport.ieSet-Cookie: ___utmvmzauvysSB=WQYQExBjiZv; path=/; Max-Age=900Set-Cookie: ___utmvazauvysSB=TjhnJRz; path=/; Max-Age=900
at io.ktor.http.cio.HttpParserKt.characterIsNotAllowed(HttpParser.kt:279)
at io.ktor.http.cio.HttpParserKt.parseHeaderValue(HttpParser.kt:261)
at io.ktor.http.cio.HttpParserKt.parseHeaders(HttpParser.kt:119)
at io.ktor.http.cio.HttpParserKt.parseResponse(HttpParser.kt:73)
at io.ktor.http.cio.HttpParserKt$parseResponse$1.invokeSuspend(HttpParser.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Process finished with exit code 1
To reproduce see attached code.
Native Client call freezes the CoroutineExceptionHandler of the coroutine it's ran from
(This issue is about Ktor 1.4.1 but I can't select it as maybe it's too early)
I have an iOS app which has been using the Ktor Client for a long time now.
As coroutines and Kotlin/Native in general isn't (weren't ?) good in multithread, this app uses a single thread.
Ktor 1.4.0 now depends on the native-mt version of coroutines, which is great, but this comes with problems.
The Ktor client mostly didn't work on Kotlin Native due to issues which have been fixed in 1.4.1.
Now, the only issue left (before the next one) preventing update from Ktor 1.3.x to 1.4.x is that Ktor 1.4 freezes the coroutine it's ran in.
I don't know why it does that and maybe there's a good reason, but I don't know how to work around that.
So the issue is, maybe there's a lack of documentation, or maybe it shouldn't freeze the coroutine in the first place unless it's needed (but I thought that kotlinx-coroutines
freezes coroutines switching thread already, so I don't understand why Ktor would do anything more than that), but right now I don't understand what I'm supposed to do to make this work. I'm attaching a small project to illustrate.
How do you make the attached project work without removing any ensureNeverFrozen()
and without dramatically changing the structure? I tried wrapping the call to HttpClient.get
in various ways (including withContext(MainScope().coroutineContext)
) but was not successful.
Here is the reproducer project: https://github.com/Ribesg/kotlin-reproducer/tree/ktor-1073
To reproduce run xcode.app
from the Xcode.
kotlin.native.concurrent.FreezingException: freezing of InvokeOnCompletion[InvokeOnCompletion@28fd2c8] has failed, first blocker is com.example.app.feature.startup.Startup@33edbf8
at 0 xcode 0x000000010a58ee8d kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23:37)
at 1 xcode 0x000000010a587d7b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44)
at 2 xcode 0x000000010a587f3b kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44)
at 3 xcode 0x000000010a5bd423 kfun:kotlin.native.concurrent.FreezingException#<init>(kotlin.Any;kotlin.Any){} + 643 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:15:9)
at 4 xcode 0x000000010a5beaa7 ThrowFreezingException + 231 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:87:15)
at 5 xcode 0x000000010a6aaf0f FreezeSubgraph + 2815
at 6 xcode 0x000000010a6c49bb Kotlin_Worker_freezeInternal + 27
at 7 xcode 0x000000010a5bd54e kfun:kotlin.native.concurrent#freeze@0:0(){0§<kotlin.Any?>}0:0 + 62 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:33:5)
at 8 xcode 0x000000010a702231 kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Boolean;kotlin.Boolean;kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 1921 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:468:54)
at 9 xcode 0x000000010a701a2f kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 207 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:452:9)
at 10 xcode 0x000000010a9320ff kfun:io.ktor.client.engine#createCallContext@io.ktor.client.engine.HttpClientEngine(kotlinx.coroutines.Job){}kotlin.coroutines.CoroutineContext + 1487 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/posix/src/io/ktor/client/engine/HttpClientEngineNative.kt:26:9)
at 11 xcode 0x000000010a8f71af kfun:io.ktor.client.engine.HttpClientEngine.$executeWithinCallContextCOROUTINE$9.invokeSuspend#internal + 1087 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/engine/HttpClientEngine.kt:77:27)
at 12 xcode 0x000000010a8f78f0 kfun:io.ktor.client.engine.HttpClientEngine.executeWithinCallContext#internal + 320 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/engine/HttpClientEngine.kt:76:21)
at 13 xcode 0x000000010a8f870a kfun:io.ktor.client.engine.HttpClientEngine.$install$lambda-0COROUTINE$10.invokeSuspend#internal + 1594 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/engine/HttpClientEngine.kt:66:32)
at 14 xcode 0x000000010a8f9127 kfun:io.ktor.client.engine.HttpClientEngine.$install$lambda-0COROUTINE$10.invoke#internal + 327 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/engine/HttpClientEngine.kt:57:64)
at 15 xcode 0x000000010a890187 kfun:io.ktor.util.pipeline.SuspendFunctionGun.$loop$<anonymous>_2COROUTINE$7.invokeSuspend#internal + 695 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:17:6)
at 16 xcode 0x000000010a8906e7 kfun:io.ktor.util.pipeline.SuspendFunctionGun.$loop$<anonymous>_2COROUTINE$7.invoke#internal + 215 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:17:4)
at 17 xcode 0x000000010a88e8f5 kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1253 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:157:99)
at 18 xcode 0x000000010a88decd kfun:io.ktor.util.pipeline.SuspendFunctionGun#proceed(){}1:0 + 413 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:113:13)
at 19 xcode 0x000000010a88e33c kfun:io.ktor.util.pipeline.SuspendFunctionGun#execute(1:0){}1:0 + 492 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:133:16)
at 20 xcode 0x000000010a8866a8 kfun:io.ktor.util.pipeline.Pipeline#execute(1:1;1:0){}1:0 + 376 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/Pipeline.kt:77:41)
at 21 xcode 0x000000010a91ac2c kfun:io.ktor.client.features.HttpSend.DefaultSender.$executeCOROUTINE$27.invokeSuspend#internal + 1404 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/features/HttpSend.kt:128:50)
at 22 xcode 0x000000010a91b270 kfun:io.ktor.client.features.HttpSend.DefaultSender.execute#internal + 320 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/features/HttpSend.kt:120:26)
at 23 xcode 0x000000010a918fb1 kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0COROUTINE$26.invokeSuspend#internal + 2273 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/features/HttpSend.kt:92:42)
at 24 xcode 0x000000010a91a0f7 kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0COROUTINE$26.invoke#internal + 327 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-client/ktor-client-core/common/src/io/ktor/client/features/HttpSend.kt:82:71)
at 25 xcode 0x000000010a890187 kfun:io.ktor.util.pipeline.SuspendFunctionGun.$loop$<anonymous>_2COROUTINE$7.invokeSuspend#internal + 695 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:17:6)
at 26 xcode 0x000000010a8906e7 kfun:io.ktor.util.pipeline.SuspendFunctionGun.$loop$<anonymous>_2COROUTINE$7.invoke#internal + 215 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:17:4)
at 27 xcode 0x000000010a88e8f5 kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1253 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:157:99)
at 28 xcode 0x000000010a88decd kfun:io.ktor.util.pipeline.SuspendFunctionGun#proceed(){}1:0 + 413 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:113:13)
at 29 xcode 0x000000010a88e0ce kfun:io.ktor.util.pipeline.SuspendFunctionGun#proceedWith(1:0){}1:0 + 222 (/Users/Aleksei.Tirman/projects/jb/ktor/ktor-utils/common/src/io/ktor/util/pipeline/SuspendFunctionGun.kt:123:16)
Android: application crashes unexpectedly when making HTTP request
To reproduce run Android application with the following code:
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val resp = makeRequest()
println(resp)
}
private fun makeRequest(): String = runBlocking(Dispatchers.Default) {
HttpClient(CIO).use { client ->
return@runBlocking client.get<String>("http://example.com")
}
}
}
As the result, the application crashes unexpectedly.
ktor-client-apache: thread stuck in ByteBufferChannel.writeAvailable
"Ktor-client-apache" #47 daemon prio=10 os_prio=0 cpu=8373041.23ms elapsed=106506.02s allocated=4716G defined_classes=12 tid=0x00007f8c9c013000 nid=0x33 runnable [0x00007f8cb55e4000]
java.lang.Thread.State: RUNNABLE
at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:2830)
at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:50)
at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
100% CPU usage on "Ktor-apache-client" threads after upgrading from 1.3.2 to 1.4.0
After updating from Ktor 1.3.2 to 1.4.0, I've noticed that some services suddenly have 100% cpu usage on some threads. It even lead to one service being completely unresponsive, as it seems like enough threads got into this broken status. Unfortunately, I was not able to get a jstack before the instance restarted, however I did get one last week, when only 1 thread was in this status. The thread dump is attached.
"Ktor-client-apache" #35 daemon prio=5 os_prio=0 tid=0x00007f995c06b000 nid=0x2a runnable [0x00007f99b62f0000]
java.lang.Thread.State: RUNNABLE
at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:50)
at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
Also attached is a screenshot of the cpu usage. It looks like it happens to one thread at a time, going up to 100% usage (this screenshot doesn't correspond to the attached dump, it was a different occurrence)
I will continue to monitor this and will be able to get more information (any suggestions?) when this happens again.
IncorrectDereferenceException thrown when ktor request call from other thread
Hello there, I see that this issue KTOR-499(https://youtrack.jetbrains.com/issue/KTOR-499) is mentioned in Ktor 1.4.1 Change log as fixed (https://github.com/ktorio/ktor/blob/master/CHANGELOG.md) and in the issue in the comments there is a statement that it is working now with the Ktor 1.4.0. I’ve forked the repository(https://github.com/RudolfHladik/ktor-mt-app), updated it to Ktor 1.4.1, Kotlin 1.4.10, Coroutines 1.3.9-native-mt and I get the IncorectDereferenceException . Is there something wrong I do or has the issue reapeared in Ktor 1.4.1?
Native: InvalidMutabilityException creating HttpClient
As of v.1.4.0, ktor client is not working on Kotlin/native iOS due to an InvalidMutabilityException.
I tested with both coroutines "1.3.9" as well as "1.3.9-native-mt", each produce the same error:
Uncaught Kotlin exception: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen kotlinx.coroutines.ChildHandleNode@3873b88
at 0 jgosdk 0x000000010b7376fd kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23:37)
at 1 jgosdk 0x000000010b73030b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44)
at 2 jgosdk 0x000000010b73055b kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44)
at 3 jgosdk 0x000000010b767deb kfun:kotlin.native.concurrent.InvalidMutabilityException#<init>(kotlin.String){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:22:60)
at 4 jgosdk 0x000000010b7682c2 ThrowInvalidMutabilityException + 690 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:92:11)
at 5 jgosdk 0x000000010b85e9ec MutationCheck + 108
at 6 jgosdk 0x000000010b9017b6 kfun:kotlinx.coroutines.internal.LinkedListNode#<set-_next>(kotlinx.coroutines.internal.LinkedListNode){} + 102 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/internal/LinkedList.kt:19:28)
at 7 jgosdk 0x000000010b901d9b kfun:kotlinx.coroutines.internal.LinkedListNode#addLast(kotlinx.coroutines.internal.LinkedListNode){} + 283 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/internal/LinkedList.kt:31:9)
at 8 jgosdk 0x000000010b9020e6 kfun:kotlinx.coroutines.internal.LinkedListNode#addOneIfEmpty(kotlinx.coroutines.internal.LinkedListNode){}kotlin.Boolean + 230 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/internal/LinkedList.kt:47:9)
at 9 jgosdk 0x000000010b8a3cbf kfun:kotlinx.coroutines.JobSupport.promoteSingleToNodeList#internal + 463 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:532:15)
at 10 jgosdk 0x000000010b8a291d kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Boolean;kotlin.Boolean;kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 2061 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:470:25)
at 11 jgosdk 0x000000010b8a208f kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 207 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:449:9)
at 12 jgosdk 0x000000010ba59d04 kfun:io.ktor.client#HttpClient(io.ktor.client.engine.HttpClientEngineFactory<0:0>;kotlin.Function1<io.ktor.client.HttpClientConfig<0:0>,kotlin.Unit>){0§<io.ktor.client.engine.HttpClientEngineConfig>}io.ktor.client.HttpClient + 1156 (/opt/buildAgent/work/a85294440dc5c6e/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClient.kt:45:36)
Circular reference for SocketException and StackOverflowError when using SLF4J logger
I upgraded my project to ktor 1.4.1 today.
I found an issue when I ran my unit test.
My logger failed to log the SocketTimeoutException.
Sample code:
fun main() {
val httpClient = HttpClient(Apache) {
install(HttpTimeout) {
socketTimeoutMillis = 3000
}
}
runBlocking {
try {
val get = httpClient.get<String>("https://httpbin.org/delay/10")
} catch (e: Exception) {
e.printStackTrace()
LoggerFactory.getLogger("main").error("test", e)
}
}
}
Output:
io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:49)
at io.ktor.client.engine.apache.ApacheHttpRequestKt$sendRequest$1.invokeSuspend(ApacheHttpRequest.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
at io.ktor.client.engine.apache.ApacheResponseConsumer.failed(ApacheResponseConsumer.kt:73)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:99)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:834)
[CIRCULAR REFERENCE:io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms]
Caused by: java.net.SocketTimeoutException: 3,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
... 11 more
Exception in thread "main" java.lang.StackOverflowError
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:590)
at java.base/java.lang.reflect.Method.invoke(Method.java:558)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:66)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:72)
Core
Gradle 5.6.4: Duplicate class kotlinx.coroutines.*
I have created a project to demonstrate the problem. https://github.com/volkovskiyda/KtorCoroutines
As you can see, there is a commit that actually fixes the problem: we should update the gradle and build tools versions.
But can we achieve successful compilation without updating these versions? There are a lot of breaking changes in the new gradle version.
Significant performance penalty with socket write() and file read() on JVM
Hi,
It seems that there are performance issues with the current implementation of TCP socket write and also with reading files. Attaching the performance comparison for ktor IO vs regular JVM raw IO:
Files:
--------------------------------------------------------------------
Ktor channel read (128MiB), 20 iterations
Avg: 1239.65ms Min: 1095ms, Max: 1723ms
Raw: 1095, 1104, 1113, 1113, 1113, 1122, 1123, 1126, 1151, 1184, 1249, 1251, 1254, 1254, 1278, 1314, 1351, 1399, 1476, 1723
--------------------------------------------------------------------
Random File Read (128MiB), 20 iterations
Avg: 75.4ms Min: 65ms, Max: 161ms
Raw: 65, 65, 66, 67, 67, 67, 67, 68, 68, 68, 68, 69, 69, 70, 70, 75, 79, 83, 96, 161
--------------------------------------------------------------------
Stream Read (128MiB), 20 iterations
Avg: 58.5ms Min: 56ms, Max: 69ms
Raw: 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 59, 59, 59, 59, 59, 65, 69
Sockets:
--------------------------------------------------------------------
JVM socket write (128MiB), 20 iterations
Avg: 87.7ms Min: 79ms, Max: 112ms
Raw: 79, 81, 81, 82, 82, 82, 83, 83, 84, 84, 84, 86, 87, 90, 90, 91, 92, 96, 105, 112
--------------------------------------------------------------------
Ktor socket write (128MiB), 20 iterations
Avg: 675.3ms Min: 572ms, Max: 968ms
Raw: 572, 582, 583, 589, 598, 612, 633, 636, 651, 676, 682, 683, 686, 688, 689, 702, 727, 763, 786, 968
You can reproduce the performance measurements with the following tests:
https://gist.github.com/Malinskiy/2dbe05a38321417fd4f23e1872593806
https://gist.github.com/Malinskiy/0172b369040cabc074ccfea9a787a79c
This is the result of an investigation of the issue raised here https://github.com/Malinskiy/marathon/issues/462
Cheers,
Anton
Input.readTextExactBytes(n) on empty input different behavior per platform
On JVM, if Input
is empty, it will throw EOFException
On Native/JS it will just return empty string
reproducer:
println(ByteReadPacket.Empty.readTextExactBytes(1).length)
JVM: fail with EOF
JS/Native: print 0
ktor-io: JVM shared function decrease performance starting from 1.4.0
While profiling rsocket-kotlin
I found in graphs/traces that one of the most frequent element are calls to SharedJVM
and boxing of Int
values. I know, that shared
delegate used in ktor-io
to support using atomics on K/N and plain value on JVM/JS. But that code optimization
isn't ok regarding to performance.
The issue consists of two parts:
- every usage of
shared
- create anonymous object on initialization - boxing/unboxing of primitives on setting/getting value
F.e.Buffer
class fromktor-io
useshared
for 4Int
fields:readPosition
,writePosition
,startGap
,limit
- and all of them are updating frequently. So all time we operate on them we will do box/unbox - performance suffer significantly...
To proof my idea, I've replaced calls of shared
to plain variables inside ktor-io
module, and built it locally. Running benchmarks I found, that it's 50% faster!
So, f.e. io.rsocket.kotlin.benchmarks.RSocketBenchmark#requestResponseParallel
benchmark on ktor version 1.4.2
gives ~200 ops/s and on local version without shared gives ~300 ops/s.
That benchmark heavily use big ByteReadPacket
s.
I think that should be fixed some how, as 50% difference is really big
Unable to catch socket exceptions
Exceptions thrown from the UDP sockets cannot be caught.
Here is a test that currently fails:
import io.ktor.network.selector.*
import io.ktor.network.sockets.*
import io.ktor.util.network.*
import io.ktor.utils.io.core.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import kotlin.io.use
import kotlin.test.*
class SocketExceptionIssue : CoroutineScope {
private val testJob = Job()
private val selector = ActorSelectorManager(Dispatchers.Default + testJob)
override val coroutineContext: CoroutineContext
get() = testJob
@AfterTest
fun tearDown() {
testJob.cancel()
selector.close()
}
@Test
fun testBroadcastFails(): Unit = runBlocking {
withTimeout(15000) {
// The following code in the assertFails lambda should throw a SocketException.
assertFails {
aSocket(selector)
.udp()
.bind {
// Do not set broadcast to true so exception is thrown.
}
.use { socket ->
// Send a broadcast which will fail because broadcasts are not enabled on this socket.
socket.send(
Datagram(
packet = buildPacket { writeText("0123456789") },
address = NetworkAddress("255.255.255.255", 56700)
)
)
}
}
}
}
}
When running this test the logs show the following:
> Task :ktor-network:jvmTest
Exception in thread "DefaultDispatcher-worker-1 @coroutine#3" java.net.SocketException: Permission denied
at java.base/sun.nio.ch.DatagramChannelImpl.send0(Native Method)
at java.base/sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:584)
at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:546)
at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:529)
at io.ktor.network.sockets.DatagramSocketImpl.sendImpl(DatagramSocketImpl.kt:98)
at io.ktor.network.sockets.DatagramSocketImpl$sender$1.invokeSuspend(DatagramSocketImpl.kt:35)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Expected an exception to be thrown, but was completed successfully.
java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
at org.junit.Assert.fail(Assert.java:88)
at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
(Coroutine boundary)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
(Coroutine creation stacktrace)
at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at io.ktor.network.sockets.tests.SocketExceptionIssue.testBroadcastFails(SocketExceptionIssue.kt:30)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
at org.junit.Assert.fail(Assert.java:88)
at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:81)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:56)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
... 50 more
io.ktor.network.sockets.tests.SocketExceptionIssue[jvm] > testBroadcastFails[jvm] FAILED
java.lang.AssertionError at SocketExceptionIssue.kt:30
Caused by: java.lang.AssertionError at SocketExceptionIssue.kt:30
1 test completed, 1 failed
You can see the SocketException in the log and that the test (with assertFails) is unable to catch it.
The following issue looks related: https://youtrack.jetbrains.com/issue/KTOR-440
Here is a fork of Ktor with the test added so you can reproduce the issue: https://github.com/Thomas-Vos/ktor/tree/socket_exception_issue (see last commit for added test). The fork is based on the 1.5.0 branch.
Upgrade kotlinx.serialization to 1.0.0-RC2
ConcurrentList.increaseCapacity() throws ArrayIndexOutOfBoundsException
I have a test that uses MockEngine which makes more than 32 calls, and then throws a java.lang.ArrayIndexOutOfBoundsException (stacktrace at the bottom).
I tracked the problem down to io.ktor.util.collections.ConcurrentList.increaseCapacity()
(ConcurrentList.kt:180):
private fun increaseCapacity(targetCapacity: Int = data.size * 2) {
val newData = SharedList<T>(targetCapacity)
for (index in 0 until targetCapacity) {
newData[index] = data[index]
}
data = newData
}
It is clear from this code that the for loop uses targetCapacity
which is going to try to access data outside of the original data size. Here is what the code should be:
private fun increaseCapacity(targetCapacity: Int = data.size * 2) {
val newData = SharedList<T>(targetCapacity)
for (index in 0 until data.size) {
newData[index] = data[index]
}
data = newData
}
Here is the stacktrace for reference.
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 32 out of bounds for length 32
at java.base/java.lang.invoke.VarHandle$1.apply(VarHandle.java:2011)
at java.base/java.lang.invoke.VarHandle$1.apply(VarHandle.java:2008)
at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:159)
at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:156)
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:62)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.lang.invoke.VarHandleObjects$Array.getVolatile(VarHandleObjects.java:438)
at java.base/java.lang.invoke.VarHandleGuards.guard_LI_L(VarHandleGuards.java:646)
at java.base/java.util.concurrent.atomic.AtomicReferenceArray.get(AtomicReferenceArray.java:100)
at io.ktor.util.collections.internal.SharedList.get(SharedList.kt:35)
at io.ktor.util.collections.ConcurrentList.increaseCapacity(ConcurrentList.kt:180)
at io.ktor.util.collections.ConcurrentList.increaseCapacity$default(ConcurrentList.kt:177)
at io.ktor.util.collections.ConcurrentList.add(ConcurrentList.kt:65)
at io.ktor.client.engine.mock.MockEngine.execute(MockEngine.kt:63)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:333)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.startCoroutineImpl(Builders.common.kt:188)
at kotlinx.coroutines.BuildersKt.startCoroutineImpl(Unknown Source)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:145)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91)
at kotlinx.coroutines.BuildersKt.async(Unknown Source)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84)
at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:124)
at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:88)
at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
at io.ktor.client.HttpClient.execute(HttpClient.kt:185)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
at com.gatsby.trade.client.OrbisOrderClient$placeOrder$$inlined$post$1.invokeSuspend(OrbisClient.kt:242)
Dependency for ktor-network 1.4.1 and 1.4.0 doesn't work on Kotlin/Multiplatform
I tried to apply the ktor-network dependency to my commonMain source set dependencies, but I received an error saying that the dependency was not found. I also tried to apply it to the dependencies of the sourceSet commonMain and it seems to work there.
Repositories applied: jcenter, mavenCentral
build.gradle.kts:
plugins {
kotlin("multiplatform") version "1.4.10"
kotlin("plugin.serialization") version "1.4.10"
}
val ktorVersion = "1.4.1"
val coroutinesVersion = "1.4.1"
val serializationVersion = "1.0.1"
allprojects {
plugins.apply("org.jetbrains.kotlin.multiplatform")
plugins.apply("org.jetbrains.kotlin.plugin.serialization")
repositories {
mavenCentral()
jcenter()
}
kotlin {
jvm {
compilations.all {
kotlinOptions.jvmTarget = "1.8"
}
testRuns["test"].executionTask.configure {
useJUnitPlatform()
}
}
val hostOs = System.getProperty("os.name")
val isMingwX64 = hostOs.startsWith("Windows")
val nativeTarget = when {
hostOs == "Mac OS X" -> macosX64("native")
hostOs == "Linux" -> linuxX64("native")
isMingwX64 -> mingwX64("native")
else -> throw GradleException("Host OS is not supported in Kotlin/Native.")
}
sourceSets {
val commonMain by getting {
dependencies {
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:$serializationVersion")
implementation("io.ktor:ktor-network:$ktorVersion")
implementation("io.ktor:ktor-io:$ktorVersion")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:$coroutinesVersion")
}
}
val commonTest by getting {
dependencies {
implementation(kotlin("test-common"))
implementation(kotlin("test-annotations-common"))
}
}
val jvmMain by getting
val jvmTest by getting {
dependencies {
implementation(kotlin("test-junit5"))
implementation("org.junit.jupiter:junit-jupiter-api:5.6.0")
runtimeOnly("org.junit.jupiter:junit-jupiter-engine:5.6.0")
}
}
val nativeMain by getting
val nativeTest by getting
}
}
}
error screenshot:
CIO client engine randomly runs into infinite loop under high load
Stuck in while (!closed)
in io.ktor.network.selector.ActorSelectorManager.process()
.
Probably related to KTOR-717.
Likely a duplicate of KTOR-531.
I also randomly get this:
Exception in thread "main" java.net.BindException: Can't assign requested address
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:503)
at java.base/sun.nio.ch.Net.connect(Net.java:492)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:751)
at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:31)
at io.ktor.network.sockets.ConnectUtilsJvmKt.connect(ConnectUtilsJvm.kt:19)
at io.ktor.network.sockets.TcpSocketBuilder.connect(TcpSocketBuilder.kt:38)
at io.ktor.client.engine.cio.ConnectionFactory.connect(ConnectionFactory.kt:24)
at io.ktor.client.engine.cio.Endpoint$connect$$inlined$repeat$lambda$1.invokeSuspend(Endpoint.kt:159)
at io.ktor.client.engine.cio.Endpoint$connect$$inlined$repeat$lambda$1.invoke(Endpoint.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:120)
at kotlinx.coroutines.TimeoutKt.withTimeoutOrNull(Timeout.kt:83)
at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:167)
at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Suppressed: java.net.BindException: Can't assign requested address
... 21 more
Repro:
import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import kotlin.random.*
import kotlin.time.*
import kotlinx.coroutines.*
@OptIn(ExperimentalTime::class)
suspend fun main() = withContext(Dispatchers.Default) {
startServer()
delay(1_000)
val batchSize = 100
val threadCount = 1
HttpClient(CIO) {
engine {
threadsCount = threadCount
}
}.use { client ->
repeat(Int.MAX_VALUE) { i ->
val time = measureTime {
coroutineScope {
repeat(batchSize) {
launch {
client.get<HttpResponse>("http://localhost:8080").readText()
}
}
}
}
println("${i + 1}: $time")
}
}
}
private fun startServer() {
embeddedServer(Netty, port = 8080) {
val random = Random(0)
val data = buildString(100_000) { repeat(100_000) { append(random.nextInt(0, 10)) } }
routing {
get("/") {
call.respondText(data, ContentType.Text.Plain)
}
}
}.start()
}
Docs
Client HttpCache feature is not documented
Wrong redirect from old documentation page
Right now the page https://ktor.io/clients/http-client/features/auth.html is redirected to https://ktor.io/docs/http-client-features.html but should be redirected to https://ktor.io/docs/features-auth.html instead.
404 page on https://ktor.io/clients/http-client.html#websockets
The first link from the comment https://github.com/ktorio/ktor/issues/507#issuecomment-410173950 gives a 404 page.
clients-index documentation page incorrect link
In this section of the doc page https://ktor.io/docs/clients-index.html#requests-responses - there are two hyperlinks; one for sending requests and one for handling responses.
The "how to receive responses" link seems correct in that it links to the response handling section of the client docs https://ktor.io/docs/response.html.
However, the "how to make requests" appears to link to a server side request handling page https://ktor.io/docs/requests.html#cookies instead of https://ktor.io/docs/request.html#specifying-a-body-for-requests.
Apache client engine is documented to support HTTP/2 but doesn't
https://ktor.io/docs/http-client-engines.html#apache
Apache is the most configurable HTTP client about right now. It supports HTTP/1.1 and HTTP/2.
Unfortunately Ktor uses httpclient 4.4 but HTTP/2 support requires httpclient 5.
https://archive.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
https://hc.apache.org/httpcomponents-client-4.4.x/index.html
Public documentation for Ktor 1.4.0 says, that it's in the development
Perhaps I am missing something, but Ktor 1.4.1 is already released.
The same time public documentation for Ktor 1.4.0 says, that it's in development.
As for me, it looks like as it's not yet ready to be used:
❗ You are viewing Ktor 1.4.0 documentation, which is still in development.
New documentation lacks artifacts information for Gradle and Maven
For example, the old documentation page about Jackson feature has ARTIFACT
block but the corresponding new documentation page has not.
Docs: Invalid id attribute syntax in html usage example
https://ktor.io/docs/html-dsl.html#basic-usage has a snippet that suggests an unsupported id assignment syntax.
h1(id = "title") {
+"Title"
}
Per kotlinx.html #66 the correct syntax appears to be to assign the id in the body.
This is with:
implementation 'org.jetbrains.kotlinx:kotlinx-html-jvm:0.7.2'
implementation 'io.ktor:ktor-html-builder:1.4.1'
Generator
Generate endpoint: 500 server errors without meaningful messages about them
- Server settings endpoint responds with lowercased ID for a build system, but
generate
endpoint expects it to be uppercased and throws an exception:
kotlinx.serialization.SerializationException: io.ktor.generator.web.json.BuildSystemType does not contain element with name 'maven'
- Server settings endpoint responds with lowercased ID for an engine, but
generate
endpoint expects it to be uppercased and throws an exception:
kotlinx.serialization.SerializationException: io.ktor.generator.web.json.Engine does not contain element with name 'netty'
Generate
endpoint responds without a message about absence of dependent feature and just throws an exception:
io.ktor.generator.core.features.MissingRequiredFeatureException: Feature auth depends on the feature routing but it is not selected
Generator API: bad error message if wrong build system or engine provided to /project/generate
If I put in JSON wrong ID for build system or engine then the server will respond with a strange error:
error: "io.ktor.generator.web.json.BuildSystemType does not contain element with name 'maven'"
I expect an error message describing that the build system or engine is unsupported and providing the supported list.
On Safari some elements have incorrect size
See attachment ("-" buttons)
Infrastructure
Publish changelog on Github releases
Release page on Github has a lot of features and tools built around them:
- RSS feed
- Email notifications for new releases
- Common API to get details about releases which works for majority open source projects on GH
Tools like dependabot and many others uses these tools to create pull request for dependecy version bump, notifications, detailed reports etc.
Pleases reconsider having detailed version changelog on Github
Enable binary-compatibility-validator in all modules
ktor npm packages haven't been published since version 1.3.2
When I go to npm.org and I search for NPM, I expect to see the latest ktor versions including version 1.4.1. Instead, I see version 1.3.2.
https://www.npmjs.com/package/ktor-ktor-client-core
Other ktor packages show this as well.
Gradle could not find artifact ktor-client-okhttp-1.4.0-samplessources.jar when okhttp specified in common sourceset
When importing implementation("io.ktor:ktor-client-okhttp:1.4.0")
, Android Studio outputs the following error:
<ij_msg_gr>Gradle import errors<ij_msg_gr><ij_nav>D:\Projekte\SteigtUm\shared-app\shared\build.gradle.kts<ij_nav><i><b>project ':shared': Unable to build Kotlin project configuration</b><eol>Details: org.gradle.internal.resolve.ArtifactNotFoundException: Could not find ktor-client-okhttp-1.4.0-samplessources.jar (io.ktor:ktor-client-okhttp:1.4.0).<eol>Searched in the following locations:<eol> https://plugins.gradle.org/m2/io/ktor/ktor-client-okhttp/1.4.0/ktor-client-okhttp-1.4.0-samplessources.jar</i>
IntelliJ IDEA Plugin
Ktor Project Creation Wizard: Generated Code For mock engine not compiling
What steps will reproduce the issue?
- Install ktor plugin
- Create project and select the mock engine feature
- Build
What is the expected result: Build compiles and runs
Build fails because of missing dependencies.
This is how it looks:
This is how I got it to compile:
I had to change the old method ("call") used at the highlighted spot, because it was still not runing or compiling since the method was marked as deprecated.
I don't know if my solution is the right one. I only start learning how to use ktor and I don't know the frameworks. For now I was only concerned with getting it compiled.
Plugin: "Could not find io.ktor:ktor-client-http-timeout" when building project created with http-timeout feature
Steps:
- Install Ktor Plugin
- Create Project With the Wizzard, like in the screenshot
- Build/Run
Expectation: Application should build an run:
Build does not work, because the ktor-client-http-timeout dependency can not be found.
I checked it on the server: https://kotlin.bintray.com/ktor/io/ktor/. I can not find this dependency there.
No changelog for 1.4.0 and 1.4.1 on the plugin page in IDE Marketplace
Samples
Routing: get matcher has higher priority than param matcher on the same level
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/378
Feel free to rename this issue, once the problem is located. Reported by @qweryty at slack:
Giving /test?error=error
and this snippet, the param
route is not handled:
fun main(args: Array<String>) {
val server = embeddedServer(Netty, 9995) {
install(DefaultHeaders)
install(Compression)
install(CallLogging)
routing {
route("test") {
param("error") {
handle {
call.respondText { "error" } // Not handled
}
}
get {
call.respondText { "test" }
}
}
}
}
server.start()
}
With the trace feature from the wip 0.9.2, this is the output:
// /, segment:0 -> SUCCESS @ /test/(method:GET))
// /test, segment:1 -> SUCCESS @ /test/(method:GET))
// /test/[error], segment:1 -> SUCCESS @ /test/[error])
// /test/(method:GET), segment:1 -> SUCCESS @ /test/(method:GET))
Even when not the same, since the previous sample should match all the methods for the param, this other sample would work (matching only get requests for the error param):
routing {
route("test") {
method(HttpMethod.Get) {
param("error") {
handle {
call.respondText { "error" } // Matched
}
}
handle {
call.respondText { "test" }
}
}
}
}
// /, segment:0 -> SUCCESS; Parameters [error=[error]] @ /test/(method:GET)/[error])
// /test, segment:1 -> SUCCESS; Parameters [error=[error]] @ /test/(method:GET)/[error])
// /test/(method:GET), segment:1 -> SUCCESS; Parameters [error=[error]] @ /test/(method:GET)/[error])
// /test/(method:GET)/[error], segment:1 -> SUCCESS @ /test/(method:GET)/[error])
[SECURITY] 0.9.* Releases were built/executed/released in the context of insecure/untrusted code
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/953
CWE-829: Inclusion of Functionality from Untrusted Control Sphere
All of these build files include resolving these dependencies over HTTP instead of HTTPS. Any of these artifacts could have been MITM to maliciously compromise them and infect the build artifacts that were produced. Additionally, if any of these JAR files were compromised, any developers using these dependencies now could continue to be infected.
- https://github.com/ktorio/ktor/blob/0.9.5/build.gradle
- https://github.com/ktorio/ktor/blob/0.9.4/build.gradle
- https://github.com/ktorio/ktor/blob/0.9.3/build.gradle
- https://github.com/ktorio/ktor/blob/0.9.2/build.gradle
- https://github.com/ktorio/ktor/blob/0.9.1/build.gradle
- https://github.com/ktorio/ktor/blob/1.0.0/build.gradle
- https://github.com/ktorio/ktor/blob/1.0.1/build.gradle
This vulnerability has a CVSS v3.0 Base Score of 8.1/10
https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator?vector=AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
This isn't just theoretical; POC code exists already to maliciously compromise jar file inflight.
See:
Please remove or update these samples, because they do not compile with new versions of Kotlin
This issue was imported from GitHub issue: https://github.com/ktorio/ktor-samples/issues/74
Especially the MPP sample. Please don't make developer's lives more difficult by letting broken code rot here on GitHub.
Form authorization not working in example (feature/auth)
This issue was imported from GitHub issue: https://github.com/ktorio/ktor-samples/issues/19
Not only are the example outdated (code is deprecated), but its also not working, thus causing quite a lot of confusion; (especially since the documentation regarding the Form authorization is pretty limited);
I can create a PR for a suitable minor change to make it work if you want :)
Need help compiling fullstack-mpp.
This issue was imported from GitHub issue: https://github.com/ktorio/ktor-samples/issues/37
I've tried to build fullstack-mpp project by opening build.gradle in IntelliJ Idea and then adding gradle.properties file with following content:
ktor_version=1.1.1
kotlin.code.style=official
kotlin_version=1.3.11
logback_version=1.2.1
When trying to run it, I get following message:
FAILURE: Build failed with an exception.
* Where:
Build file 'C:\Users\czare\IdeaProjects\fullstack-mpp\build.gradle' line: 81
* What went wrong:
Could not determine the dependencies of task ':run'.
> No such property: files for class: org.jetbrains.kotlin.gradle.plugin.mpp.DefaultKotlinCompilationOutput
I didn't change anything in build.gradle file. What am I doing wrong?
Missing dependency io.ktor:ktor-client-ios_debug_ios_arm64:1.1.1
This issue was imported from GitHub issue: https://github.com/ktorio/ktor-samples/issues/35
Using commit 5480a4d8eb338302f181c054b2a067e67ff7e366
- git clone
- ./gradlew build
- Error:
Could not resolve all files for configuration ':client-mpp:iosArm64CompileKlibraries'.
Could not find io.ktor:ktor-client-ios_debug_ios_arm64:1.1.1.
Server
Configure feature multiple times
What do you think about allowing every feature to be installed/configured multiple times?
eg
install(StatusPages) {
exception<IllegalArgumentException> {
call.respond(HttpStatusCode.InternalServerError)
}
}
// In another module
install(StatusPages) {
exception<NotImplementedError> {
call.respond(HttpStatusCode.Teapot)
}
}
Currently, configuring the Routing
feature multiple times is only possible with the Application.routing
extension, because it uses featureOrNull(Routing)?.apply(configuration) ?: install(Routing, configuration)
under the hood
Support application/xml as part for the ContentNegotiation feature
In addition to support application/json: https://ktor.io/features/jackson.html
Ktor stopped working with latest Tomcat 9.0.39
Add UUID to DefaultConversionService
If you want to use an UUID as a parameter in a request, eg /users/{UUID}
, it is currently not possible to convert the parameter into an UUID.
val id: UUID by call.parameters
Exception: io.ktor.util.DataConversionException: Type class java.util.UUID is not supported in default data conversion service
Serializing collections of different element types
Original question from https://stackoverflow.com/questions/64319112/why-serializing-collections-of-different-element-types-is-not-supported-in-ktor
To reproduce use the following code example:
@Serializable
sealed class Message {
abstract val content: String
}
@Serializable
data class BroadcastMessage(override val content: String) : Message()
@Serializable
data class DirectMessage(override val content: String, val recipient: String) : Message()
val data: List<Message> = listOf(
DirectMessage("Hey, Joe!", "Joe"),
BroadcastMessage("Hey, all!")
)
fun main() {
embeddedServer(Netty, port = 8080, host = "127.0.0.1") {
install(ContentNegotiation) {
json()
}
routing {
get("/") {
call.respond(data)
}
}
}.start(wait = true)
}
Result:
java.lang.IllegalStateException: Serializing collections of different element types is not yet supported. Selected serializers: [DirectMessage, BroadcastMessage]
at io.ktor.serialization.SerializerLookupKt.elementSerializer(SerializerLookup.kt:71)
at io.ktor.serialization.SerializerLookupKt.serializerForSending(SerializerLookup.kt:41)
at io.ktor.serialization.SerializationConverter.convertForSend(SerializationConverter.kt:114)
at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:149)
at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at ServerKt$main$1$2$1.invokeSuspend(server.kt:39)
at ServerKt$main$1$2$1.invoke(server.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.routing.Routing.executeResult(Routing.kt:148)
at io.ktor.routing.Routing.interceptor(Routing.kt:35)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Nested Authentication: Only first authentication is executed
Hey,
To have a fine control over resources for different audiences (eg public, logged-in user, admin), you use the Route.authenticate(...)
function.
open class User(val name: String) : Principal
class Admin(name: String) : User(name)
Unfortunately, only the first Route.authenticate(...)
function is called.
This results into the execution of the second route (/login/admin
) with a wrong User
principal and not with a Admin
principal.
Is this the wanted design?
fun Application.module() {
authentication {
basic("myBasicAuth") {
realm = "Ktor Server"
validate {
when (it.name) {
"user" -> User(it.name)
"admin" -> Admin(it.name)
else -> null
}
}
}
}
authentication {
basic("myAdminAuth") {
realm = "Ktor Server"
validate {
when (it.name) {
"admin" -> Admin(it.name)
else -> null
}
}
}
}
routing {
get("/") {
call.respondText { "Hello 🌍" }
}
authenticate("myBasicAuth") {
route("/login") {
get {
val principal = call.principal<User>()!!
call.respondText("Hello ${principal.name}")
}
authenticate("myAdminAuth") {
get("/admin") {
val principal = call.principal<Admin>()!!
call.respondText("Hello ${principal.name}")
}
}
}
}
}
}
Testproject: https://github.com/hfhbd/ktor-auth-bug
Request headers exceeding expected threshold are not handled correctly
Kotlin version: 1.3.70
Ktor version: 1.3.2
In situations when receiving large cookies and the headers exceed the 8K limit (using Netty as engine), not all headers get loaded.
Example 1:
- When we have the
ContentNegotiation
feature enabled - And a
content-type: application/json
is present on the request - And the cookie header is very large (the sum of all headers exceeding 8K)
- Then the
ContentNegotiation
feature resolves the content type toAny
(*/*
) - probably because it can't load all headers - This results in a situation where the server responds with
415 - Unsupported Media Type
.
In this particular example probably a 431 - Requests header fields too large
is more appropriate.
Example 2:
- Consider the hypothesis from Example 1
- When the
content-type: application/json
header is loaded before the cookie header - Then the request passes successfully through the
ContentNegotiation
feature - But the request hangs
This seems like an issue where Ktor does not correctly interpret errors from the Netty engine.
Sample engine error:
DefaultHttpRequest(decodeResult: failure(io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.), version: HTTP/1.1)
POST /v1/api HTTP/1.1
Host: localhost:8080
Accept: */*
cache-control: no-cache
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
content-type: text/plain;charset=UTF-8
Sessions + SSL (Netty)
edit: https://github.com/dm6801/ktor_sessions_ssl
when trying to use Session feature with cookies
install(Sessions) {
cookie<MySession>("MY_SESSION")
}
accessing https:/localhost:443/ throws:
java.lang.UnsupportedOperationException: null
(Coroutine boundary)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:55)
(Coroutine creation stacktrace)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:188)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:38)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:821)
Caused by: java.lang.UnsupportedOperationException: null
at io.ktor.server.netty.http2.NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64)
at io.ktor.sessions.SessionTransportCookie.receive(SessionTransportCookie.kt:28)
at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:204)
at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:59)
at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
relevant line NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64):
override val cookies: RequestCookies
get() = throw UnsupportedOperationException()
- http://localhost:8080/ - works
Allow any header in CORS configuration
Motivation from the PR: https://github.com/ktorio/ktor/pull/2034
I would like to support custom headers in a rest api called from a web browser, that I cannot know at application load.
I would therefore like to be able to mirror the allowed headers as requested in the preflight call, I believe this functionality is implemented in existing frameworks: for example expressjs
https://expressjs.com/en/resources/middleware/cors.html
"allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex:Content-Type,Authorization
) or an array (ex:['Content-Type', 'Authorization']
). If not specified, defaults to reflecting the headers specified in the request’sAccess-Control-Request-Headers
header."
Netty: NumberFormatException when make a request via HTTP/2 without explicit port
Hello.
There is a bug with Http2LocalConnectionPoint. If there is no port specified in authority then it tries to parse domain as port.
I've created a project to reproduce the bug: https://github.com/Vlad8161/netty-bug-sample
To reproduce it run the server on 443 port and access it without explicit port specifying.
Stack Trace:
java.lang.NumberFormatException: For input string: "localhost"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at io.ktor.server.netty.http2.Http2LocalConnectionPoint.getPort(Http2LocalConnectionPoint.kt:31)
at io.ktor.features.HSTS.intercept(HSTS.kt:87)
at io.ktor.features.HSTS$Feature$install$1.invokeSuspend(HSTS.kt:101)
at io.ktor.features.HSTS$Feature$install$1.invoke(HSTS.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:101)
at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:189)
at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:100)
at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:140)
at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139)
at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.startCoroutineImpl(Builders.common.kt:182)
at kotlinx.coroutines.BuildersKt.startCoroutineImpl(Unknown Source)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:145)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:54)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
The problem is at Http2LocalConnectionPoint:31
I think is would be better to do something like this:
nettyHeaders.authority()?.toString()?.substringAfter(":", if (scheme == "https") "443" else "80")?.toInt()
because often local port information is not enough
Thanks in advance :)
Exception after WebSocketSession.close() invocation.
Invoke WebSocketSession.close()
It causes the following exception:
kotlinx.coroutines.experimental.channels.ClosedSendChannelException: Channel was closed
at kotlinx.coroutines.experimental.channels.Closed.getSendException(AbstractChannel.kt:1008)
at kotlinx.coroutines.experimental.channels.AbstractSendChannel.offer(AbstractChannel.kt:185)
at kotlinx.coroutines.experimental.channels.AbstractSendChannel.send(AbstractChannel.kt:175)
at kotlinx.coroutines.experimental.channels.ChannelCoroutine.send$suspendImpl(ChannelCoroutine.kt:33)
at kotlinx.coroutines.experimental.channels.ChannelCoroutine.send(ChannelCoroutine.kt)
at kotlinx.coroutines.experimental.channels.LazyActorCoroutine.send(Actor.kt:192)
at io.ktor.http.cio.websocket.WebSocketWriter.flush(WebSocketWriter.kt:122)
at io.ktor.http.cio.websocket.RawWebSocket.flush(RawWebSocket.kt:37)
at io.ktor.websocket.DelegatedWebSocketServerSession.flush(WebSocketServerSession.kt)
at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl.flush(DefaultWebSocketSessionImpl.kt)
at io.ktor.websocket.DelegatedDefaultWebSocketServerSession.flush(WebSocketServerSession.kt)
at io.ktor.http.cio.websocket.WebSocketSessionKt.close(WebSocketSession.kt:58)
WebSocketSession.close invokes flush that fails because the websocket is already closed at the moment.
Multiple matching paths do not choose higher quality path (component with more static values) when the lower quality one is authenticated
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1158
Ktor Version
1.1.3
Ktor Engine Used(client or server and name)
Jetty server
JVM Version, Operating System and Relevant Context
JDK 11, Linux 5.0
Feedback
Given the following code:
import io.ktor.application.call
import io.ktor.response.respondText
import io.ktor.routing.get
import io.ktor.routing.routing
import io.ktor.server.engine.embeddedServer
import io.ktor.server.jetty.Jetty
fun main() {
val server = embeddedServer(Jetty, port = 9010) {
routing {
get("/foo:{bar}/{baz}") {
call.respondText("Received route1 bar=${call.parameters["bar"]}, baz=${call.parameters["baz"]}")
}
get("/{bar}/{baz}") {
call.respondText("Received route2 bar=${call.parameters["bar"]}, baz=${call.parameters["baz"]}")
}
}
}
server.start(wait = true)
}
The first route has a static component beginning with /foo:
whereas the second route is all dynamic parameters. The first route is therefore of higher quality for matching purposes.
With this call:
http get :9010/foo:a/b
the result is as expected: Received route1 bar=a, baz=b
However, when changing the code to wrap the second route in an authenticate
block like this:
routing {
get("/foo:{bar}/{baz}") {
call.respondText("Received route1 bar=${call.parameters["bar"]}, baz=${call.parameters["baz"]}")
}
authenticate {
get("/{bar}/{baz}") {
call.respondText("Received route2 bar=${call.parameters["bar"]}, baz=${call.parameters["baz"]}")
}
}
}
I would expect the routing decision to be the same, but:
http get :9010/foo:a/b
now attempts to execute the lower quality authenticated path:
HTTP/1.1 401 Unauthorized
Wrong session id get stuck at clients
Currently, if a client keeps wrong/outdated session ID then there is no way to make it forget it. Because of it, a client is sending the wrong ID, again and again, causing a lot of lookup errors at the server. On debugging log-level, it produces a lot of error messages.
Like other servers, we could expire the corresponding cookie
Confusing log message about failed session lookup
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/445
Hello,
I have a problem when I do : call.sessions.get<MySession>() with a wrong header. I have the following error 2018-06-18 15:59:07.681 [nettyCallPool-4-1] DEBUG Application - Failed to lookup session: java.util.NoSuchElementException: No session data found for id 684eb3846f0441113ab51e24471b8062f1d30f5290a11046d17d5725470fdce7.
"For unexistant sessions that are not available, would be like if no session was provided and you can do whatever you need in your route handler"
Thank for your help !
MultiPartData.readAllParts() throws java.io.IOException when multipart list is empty
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/482
Problem description: When receiving an empty multipart-form request on a POST route, the request handler throws an IOException, preventing graceful handling or the display of an error message. Instead, a 500 error message is returned.
Minimum working example:
import io.ktor.application.*
import io.ktor.content.readAllParts
import io.ktor.response.*
import io.ktor.request.*
import io.ktor.routing.post
import io.ktor.routing.routing
fun main(args: Array<String>): Unit = io.ktor.server.netty.DevelopmentEngine.main(args)
fun Application.module() {
routing {
post("/add") {
if(call.request.isMultipart()) {
val multipart = call.receiveMultipart()
val formItems = multipart.readAllParts() // <-- this breaks when sent an empty request
call.respondText("recvd: $multipart")
}
else {
call.respond("no multipart, fix request!")
}
}
}
}
Send a request with an empty multipart body to see the issue:
curl -X POST \
http://localhost:8080/add \
-H 'Cache-Control: no-cache' \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'
Error Message:
22:57:11.866 [nettyCallPool-4-4] ERROR Application - Unhandled: POST - /add
java.io.IOException: Broken delimiter occurred
at kotlinx.coroutines.experimental.io.DelimitedKt$skipDelimiterSuspend$2.doResume(Delimited.kt:57)
at kotlinx.coroutines.experimental.io.DelimitedKt$skipDelimiterSuspend$2.invoke(Delimited.kt)
at kotlinx.coroutines.experimental.io.DelimitedKt$skipDelimiterSuspend$2.invoke(Delimited.kt)
at kotlinx.coroutines.experimental.io.ByteBufferChannel.lookAheadSuspend(ByteBufferChannel.kt:1746)
at kotlinx.coroutines.experimental.io.DelimitedKt.skipDelimiterSuspend(Delimited.kt:55)
at kotlinx.coroutines.experimental.io.DelimitedKt.skipDelimiter(Delimited.kt:50)
at io.ktor.http.cio.MultipartKt.boundary(Multipart.kt:89)
at io.ktor.http.cio.MultipartKt$parseMultipart$1.doResume(Multipart.kt:158)
at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resume(CoroutineImpl.kt:42)
at kotlinx.coroutines.experimental.DispatchedKt.resumeCancellable(Dispatched.kt:209)
at kotlinx.coroutines.experimental.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:35)
at kotlinx.coroutines.experimental.CoroutineStart.invoke(CoroutineStart.kt:111)
at kotlinx.coroutines.experimental.AbstractCoroutine.start(AbstractCoroutine.kt:165)
at kotlinx.coroutines.experimental.channels.ProduceKt.produce(Produce.kt:95)
at kotlinx.coroutines.experimental.channels.ProduceKt.produce$default(Produce.kt:88)
at io.ktor.http.cio.MultipartKt.parseMultipart(Multipart.kt:145)
at io.ktor.http.cio.MultipartKt.parseMultipart(Multipart.kt:138)
at io.ktor.http.cio.CIOMultipartDataBase.<init>(CIOMultipartData.kt:33)
at io.ktor.http.cio.CIOMultipartDataBase.<init>(CIOMultipartData.kt:31)
at io.ktor.server.engine.DefaultTransformKt.multiPartData(DefaultTransform.kt:70)
at io.ktor.server.engine.DefaultTransformKt.access$multiPartData(DefaultTransform.kt:1)
at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$2.doResume(DefaultTransform.kt:33)
at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$2.invoke(DefaultTransform.kt)
at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$2.invoke(DefaultTransform.kt)
at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
at io.ktor.request.ApplicationReceiveFunctionsKt.receive(ApplicationReceiveFunctions.kt:64)
at io.sebi.ApplicationKt$module$1$1.doResume(application.kt:31)
at io.sebi.ApplicationKt$module$1$1.invoke(application.kt)
at io.sebi.ApplicationKt$module$1$1.invoke(application.kt)
at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
at io.ktor.routing.Routing.executeResult(Routing.kt:100)
at io.ktor.routing.Routing.interceptor(Routing.kt:25)
at io.ktor.routing.Routing$Feature$install$1.doResume(Routing.kt:66)
at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt:51)
at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.doResume(DefaultEnginePipeline.kt:66)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.doResume(NettyApplicationCallHandler.kt:31)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt:10)
at kotlinx.coroutines.experimental.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:44)
at kotlinx.coroutines.experimental.CoroutineStart.invoke(CoroutineStart.kt:113)
at kotlinx.coroutines.experimental.AbstractCoroutine.start(AbstractCoroutine.kt:165)
at kotlinx.coroutines.experimental.BuildersKt__Builders_commonKt.launch(Builders.common.kt:72)
at kotlinx.coroutines.experimental.BuildersKt.launch(Unknown Source)
at kotlinx.coroutines.experimental.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:64)
at kotlinx.coroutines.experimental.BuildersKt.launch$default(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:22)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:16)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:844)
Netty: Not started servers leak resources
Netty.create()
does some heavy initialization immediately, for example creating EventLoopGroup
s. These in turn create pipes.
Even though the engine is never started it allocates resources and would only release them when calling stop()
.
This was quite confusing in unit testing where I create engines repeatedly. They may or may not get started and only get stopped if they were actually started.
I'd expect resources to be only allocated when I call start()
, esp. since only stop()
will release them.
repeat(1000) {
embeddedServer(Netty, applicationEngineEnvironment { })
}
Exception in thread "main" java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:96)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:91)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
at io.ktor.server.netty.EventLoopGroupProxy$Companion.create(NettyApplicationEngine.kt:190)
at io.ktor.server.netty.NettyApplicationEngine.<init>(NettyApplicationEngine.kt:90)
at io.ktor.server.netty.Netty.create(Embedded.kt:14)
at io.ktor.server.netty.Netty.create(Embedded.kt:12)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:79)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:77)
at server.MainKt.main(Main.kt:15)
at server.MainKt$$$main.invoke(Unknown Source)
at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt$createCoroutineUnintercepted$$inlined$createCoroutineFromSuspendFunction$IntrinsicsKt__IntrinsicsJvmKt$1.invokeSuspend(IntrinsicsJvm.kt:205)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlin.coroutines.ContinuationKt.startCoroutine(Continuation.kt:115)
at kotlin.coroutines.jvm.internal.RunSuspendKt.runSuspend(RunSuspend.kt:19)
at server.MainKt.main(Main.kt)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
... 19 more
Caused by: java.io.IOException: Too many open files
at java.base/sun.nio.ch.IOUtil.makePipe(Native Method)
at java.base/sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:86)
at java.base/sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:35)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
... 23 more
SerializationException when type of serializable class property is generic
To reproduce, run the following code and make a request to the /bad
endpoint:
@Serializable
class ApiResponse<T>(
val result: T? = null,
)
fun main() {
embeddedServer(Netty, port = 8090, host = "127.0.0.1") {
install(ContentNegotiation) {
json()
}
routing {
get("/bad") {
call.respond(ApiResponse(result = "asd"))
}
}
}.start(wait = true)
}
The unexpected exception is logged:
kotlinx.serialization.SerializationException: Serializer for class 'ApiResponse' is not found.
Mark the class as @Serializable or provide the serializer explicitly.
at kotlinx.serialization.internal.Platform_commonKt.serializerNotRegistered(Platform.common.kt:104)
at kotlinx.serialization.SerializersKt__SerializersKt.serializer(Serializers.kt:127)
at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
at io.ktor.serialization.SerializerLookupKt.serializerForSending(SerializerLookup.kt:59)
at io.ktor.serialization.SerializationConverter.convertForSend(SerializationConverter.kt:114)
at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:149)
at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at MainKt$main$1$2$1.invokeSuspend(main.kt:30)
at MainKt$main$1$2$1.invoke(main.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.routing.Routing.executeResult(Routing.kt:148)
at io.ktor.routing.Routing.interceptor(Routing.kt:35)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Also, I am able to reproduce it in 1.4.2-SNAPSHOT
(the latest master branch).
Add excludeSuffix to HttpsRedirect feature
Test Infrastructure
Inconsistent TestApplicationRequest and Client HttpRequestBuilder API's
When creating a server test with withTestApplication
and handleRequest
, the APIs from TestApplicationRequest
differs from the "real-world" client APIs defined in HttpRequestBuilder
.
For consistency and reusing the test code, these APIs should match.
// Client (JS)
HttpClient().get<String>("$baseURL/login") {
header(HttpHeaders.Authorization, "Basic ${"admin:password".encodeBase64()}")
}
// Server JVM (Test)
handleRequest(HttpMethod.Post, "/login") {
addHeader(HttpHeaders.Authorization, "Basic ${"admin:password".encodeBase64()}")
}
Replace JUnit with kotlin.test
Replace JUnit with kotlin.test as much as possible
Reason
- Better Kotlin support, like assertEquals
- theoretical platform independent
Other
UDP sockets on native
We would like to use UDP sockets in our iOS app to control LIFX lights. We are currently using the Android UDP sockets of Ktor and they work great. However, when trying to use the UDP sockets on iOS, the following exception is thrown: UDP sockets are not supported
I have been working on an implementation for UDP sockets for native, see: https://github.com/ktorio/ktor/pull/2129
Getting UnsupportedOperationException when calling call.receive
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1249
Ktor pre-release version 1.2.3-rc
When trying to parse the request using:
suspend inline fun <reified T : Any> ApplicationCall.receive(): T = receive(T::class)
In this way:
val obj = call.receive<MyCustomObject>()
It produces de following exception:
java.lang.UnsupportedOperationException: This function has a reified type parameter and thus can only be inlined at compilation time, not called directly.
at kotlin.jvm.internal.Intrinsics.throwUndefinedForReified(Intrinsics.java:193)
at kotlin.jvm.internal.Intrinsics.throwUndefinedForReified(Intrinsics.java:187)
at kotlin.jvm.internal.Intrinsics.reifiedOperationMarker(Intrinsics.java:197)
at "my package"$main$2$3.invokeSuspend(KtorServices.kt:205)
at "my package"$main$2$3.invoke(KtorServices.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:268)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:141)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:161)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:27)
at io.ktor.routing.Routing.executeResult(Routing.kt:147)
at io.ktor.routing.Routing.interceptor(Routing.kt:34)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:99)
at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:268)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:141)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:106)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:268)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:141)
at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139)
at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:268)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:141)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:161)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:27)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:268)
at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:141)
at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:161)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:27)
at io.ktor.server.servlet.KtorServlet$blockingService$1.invokeSuspend(KtorServlet.kt:148)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedKt.resumeCancellable(Dispatched.kt:447)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:25)
at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:109)
at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:154)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:53)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at io.ktor.server.servlet.KtorServlet.blockingService(KtorServlet.kt:113)
at io.ktor.server.servlet.KtorServlet.service(KtorServlet.kt:71)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:119)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:187)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:293)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:213)
at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:722)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:685)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:655)
at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:847)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:270)
at java.lang.Thread.run(Thread.java:748)
HttpClient can only be used on the main thread for native targets
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1559
Ktor Version and Engine Used (client or server and name)
Ktor client 1.2.6, default platform engine (iOS)
Describe the bug
HttpClient can only be used from the main thread. It does not matter on which thread the object is instantiated.
The issue is caused by global declarations that are not annotated properly for Kotlin/Native.
Uncaught Kotlin exception: io.ktor.http.URLParserException: Fail to parse url: <redacted>
Caused by: kotlin.native.IncorrectDereferenceException: Trying to access top level value not marked as @ThreadLocal or @SharedImmutable from non-main thread
at 0 test.kexe 0x0000000101edb8e7 kfun:kotlin.Throwable.<init>(kotlin.String?)kotlin.Throwable + 87 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Throwable.kt:22:37)
at 1 test.kexe 0x0000000101ed4ee5 kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44)
at 2 test.kexe 0x0000000101ed4aa5 kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44)
at 3 test.kexe 0x0000000101f07b55 kfun:kotlin.native.IncorrectDereferenceException.<init>(kotlin.String)kotlin.native.IncorrectDereferenceException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/Runtime.kt:30:36)
at 4 test.kexe 0x0000000101f25e99 ThrowIncorrectDereferenceException + 137 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/internal/RuntimeUtils.kt:87:11)
at 5 test.kexe 0x0000000102200ed9 CheckIsMainThread + 25
at 6 test.kexe 0x00000001020e16a3 kfun:io.ktor.http.<get-URL_ALPHABET_CHARS>#internal + 35 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt:11:9)
at 7 test.kexe 0x00000001020e2ecd kfun:io.ktor.http.encodeURLPath@kotlin.String.()kotlin.String + 717 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt:59:0)
...
I am using the CoroutineWorker library from Autodesk.
To Reproduce
Steps to reproduce the behavior:
- Make a client request from a non-main thread for a native target.
`ByteBufferChannel.readRemaining` doesn't read whole channel
Fix HTTP2 tests with new versions of java8
Fix typo `val socketTimeout` in `CIOEngineConfig` cause it's a property in the config
Blocking calls (ReentrantLock.lock) on Apache engine dispatcher threads
I'm using BlockHound to check if there are any blocking calls within my coroutines that aren't using IO dispatcher.
BlockHound reports the following occasion for Ktor client with Apache engine on thread ktor-apache-dispatcher-worker-4
.
Maybe it's a bug, maybe it's a misconfiguration of the dispatcher threads as CPU-bound.
reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:68)
at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:33)
at io.ktor.client.engine.apache.ApacheEngine.execute(ApacheEngine.kt:37)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Release 1.4.1
Fix InvalidMutabilityException for tests in master
Native initialisation order leads to fail in constructor when we use this
in field initialization in shared due to freeze:
kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.util.collections.internal.SharedForwardList@9fa908
at kfun:kotlin.Throwable#<init>(kotlin.String?){} (0x2dc9b7)
at kfun:kotlin.Exception#<init>(kotlin.String?){} (0x2d6b65)
at kfun:kotlin.RuntimeException#<init>(kotlin.String?){} (0x2d67c5)
at kfun:kotlin.native.concurrent.InvalidMutabilityException#<init>(kotlin.String){} (0x303585)
at ThrowInvalidMutabilityException (0x304c53)
at MutationCheck (0x5a519e)
at kfun:io.ktor.util.collections.internal.SharedForwardList#<init>(){} (0x4ff407)
at kfun:io.ktor.util.collections.ConcurrentMap#<init>(io.ktor.util.Lock;kotlin.Int){} (0x4e9814)
at kfun:io.ktor.util.collections.ConcurrentMap#<init>(io.ktor.util.Lock?;kotlin.Int;kotlin.Int;kotlin.native.internal.DefaultConstructorMarker?){} (0x4e9af2)
at kfun:io.ktor.util.ConcurrentMapTest#testEmpty(){} (0x52e1a5)
at kfun:io.ktor.util.$ConcurrentMapTest$test$0.$testEmpty$FUNCTION_REFERENCE$14.invoke#internal (0x52f893)
at kfun:io.ktor.util.$ConcurrentMapTest$test$0.$testEmpty$FUNCTION_REFERENCE$14.$<bridge-UNNN>invoke(-1:0){}#internal (0x52f909)
at kfun:kotlin.native.internal.test.BaseClassSuite.TestCase#run(){} (0x32755b)
at kfun:kotlin.native.internal.test.TestRunner.run#internal (0x3215b8)
at kfun:kotlin.native.internal.test.TestRunner.runIteration#internal (0x32276e)
at kfun:kotlin.native.internal.test.TestRunner#run(){}kotlin.Int (0x3230a5)
at kfun:kotlin.native.internal.test#testLauncherEntryPoint(kotlin.Array<kotlin.String>){}kotlin.Int (0x316eec)
at kfun:kotlin.native.internal.test#main(kotlin.Array<kotlin.String>){} (0x316db7)
at Konan_start (0x31701b)
There is also usages of simple references instead of shared in tests and client capabilities.
Empty body in response using macosx64 target
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1622
Ktor Version and Engine Used (client or server and name)
1.3.0, client (macosx64) (using curl)
Describe the bug
When using macosx64 target in a Multiplatform project I'm getting empty BODY in response
HttpClient: REQUEST: http://api.open-notify.org/astros.json
HttpClient: METHOD: HttpMethod(value=GET)
HttpClient: COMMON HEADERS
HttpClient: -> Accept: application/json
HttpClient: -> Accept-Charset: UTF-8
HttpClient: CONTENT HEADERS
HttpClient: BODY Content-Type: null
HttpClient: BODY START
HttpClient: BODY END
The same code for iOS logs
HttpClient: REQUEST: http://api.open-notify.org/astros.json
HttpClient: METHOD: HttpMethod(value=GET)
HttpClient: COMMON HEADERS
HttpClient: -> Accept: application/json
HttpClient: -> Accept-Charset: UTF-8
HttpClient: CONTENT HEADERS
HttpClient: BODY Content-Type: null
HttpClient: BODY START
HttpClient: BODY END
HttpClient: BODY Content-Type: application/json
HttpClient: BODY START
HttpClient: {"people": [{"craft": "ISS", "name": "Andrew Morgan"}, {"craft": "ISS", "name": "Oleg Skripochka"}, {"craft": "ISS", "name": "Jessica Meir"}], "message": "success", "number": 3}
HttpClient: BODY END
The following is the common code that uses ktor
class PeopleInSpaceApi {
private val url = "http://api.open-notify.org/astros.json"
private val client by lazy {
HttpClient() {
install(JsonFeature) {
serializer = KotlinxSerializer(Json(JsonConfiguration(strictMode = false)))
}
install(Logging) {
logger = Logger.DEFAULT
level = LogLevel.ALL
}
}
}
suspend fun fetchPeople(): AstroResult {
return client.get(url)
}
}
I'm using following dependencies
macOSMain.dependencies {
// Coroutines
implementation('org.jetbrains.kotlinx:kotlinx-coroutines-core-macosx64') {
version {
strictly '1.3.3-native-mt'
}
}
// Ktor
implementation "io.ktor:ktor-client-curl:${Versions.ktor}"
implementation "io.ktor:ktor-client-core-macosx64:${Versions.ktor}"
implementation "io.ktor:ktor-client-json-macosx64:${Versions.ktor}"
implementation "io.ktor:ktor-client-logging-macosx64:${Versions.ktor}"
implementation "io.ktor:ktor-client-serialization-macosx64:${Versions.ktor}"
// Serialize
implementation "org.jetbrains.kotlinx:kotlinx-serialization-runtime-macosx64:${Versions.kotlinxSerialization}"
}
(kotlinxSerialization = "0.14.0")
MalformedInputException: Input length = 1, when uploading an image multipart
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1766
Ktor Version and Engine Used (client or server and name)
Ktor 1.3.2, Kotlin 1.3.70, Netty server, ContentNegotiation, Jackson parser, Exposed.
I simply used the Ktor project generator for Gradle, and checked a few boxes for modules as mentioned above. After which I managed to connect a database & a few endpoints, but my core endpoint for uploading images doesn't work.
Describe the bug
When trying to upload a simple image using Retrofit & Multipart support in Android, I get a MalformedInputException on my server.
Retrofit/Android code:
@Multipart
@POST("/api/upload")
suspend fun uploadImages(@Part("image") imageFile: RequestBody): UploadResponse
Upload code:
val requestBody: RequestBody = MultipartBody.Builder()
.setType(MultipartBody.FORM)
.addFormDataPart("title", "Image Upload")
.addFormDataPart(
"image", // part name
file.path.split("/").last(), // file name
file.asRequestBody("image/jpg".toMediaType()) // Content-Type: image/jpg
)
.build()
return apiService.uploadImages(requestBody)
Using the new way of creating RequestBody objects in OkHttp. I followed the documentation to receive the multipart data, and I've formed a MalformedInputException.
To Reproduce
Steps to reproduce the behavior:
- Use Retrofit to Upload an image to a Ktor server.
- call
receiveMultipart()
to receive MP on the Ktor side. - See error
Expected behavior
Multipart data should be received correctly.
Stack trace:
[nioEventLoopGroup-4-1] ERROR Application - Unhandled: POST - /api/upload
io.ktor.utils.io.charsets.MalformedInputException: Input length = 1
at io.ktor.utils.io.charsets.CharsetJVMKt.throwExceptionWrapped(CharsetJVM.kt:323)
at io.ktor.utils.io.charsets.CharsetJVMKt.decodeImplSlow(CharsetJVM.kt:289)
at io.ktor.utils.io.charsets.CharsetJVMKt.decodeExactBytes(CharsetJVM.kt:254)
at io.ktor.utils.io.core.StringsKt.readTextExactBytes(Strings.kt:293)
at io.ktor.utils.io.core.StringsKt.readTextExactBytes$default(Strings.kt:292)
at io.ktor.utils.io.core.AbstractInput.readText(AbstractInput.kt:468)
at io.ktor.utils.io.core.AbstractInput.readText$default(AbstractInput.kt:465)
at io.ktor.http.cio.CIOMultipartDataBase.partToData(CIOMultipartData.kt:127)
at io.ktor.http.cio.CIOMultipartDataBase$partToData$1.invokeSuspend(CIOMultipartData.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:47