abstract class StatefulProcessorWithInitialState[K, I, O, S] extends StatefulProcessor[K, I, O]
Stateful processor with support for specifying initial state. Accepts a user-defined type as initial state to be initialized in the first batch. This can be used for starting a new streaming query with existing state from a previous streaming query.
- Annotations
- @Evolving()
- Source
- StatefulProcessor.scala
- Alphabetic
- By Inheritance
- StatefulProcessorWithInitialState
- StatefulProcessor
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new StatefulProcessorWithInitialState()
Abstract Value Members
- abstract def handleInitialState(key: K, initialState: S, timerValues: TimerValues): Unit
Function that will be invoked only in the first batch for users to process initial states.
Function that will be invoked only in the first batch for users to process initial states. The provided initial state can be arbitrary dataframe with the same grouping key schema with the input rows, e.g. dataframe from data source reader of existing streaming query checkpoint.
Note that in microbatch mode, this function can be called for one or more times per grouping key. If the grouping key is not seen within the initial state dataframe rows, then the function will not be invoked for that key.
- key
\- grouping key
- initialState
\- A row in the initial state to be processed
- timerValues
\- instance of TimerValues that provides access to current processing/event time if available
- abstract def handleInputRows(key: K, inputRows: Iterator[I], timerValues: TimerValues): Iterator[O]
Function that will allow users to interact with input data rows along with the grouping key and current timer values and optionally provide output rows.
Function that will allow users to interact with input data rows along with the grouping key and current timer values and optionally provide output rows.
Note that in microbatch mode, input rows for a given grouping key will be provided in a single function invocation. If the grouping key is not seen in the current microbatch, this function will not be invoked for that key.
- key
\- grouping key
- inputRows
\- iterator of input rows associated with grouping key
- timerValues
\- instance of TimerValues that provides access to current processing/event time if available
- returns
\- Zero or more output rows
- Definition Classes
- StatefulProcessor
- abstract def init(outputMode: OutputMode, timeMode: TimeMode): Unit
Function that will be invoked as the first method that allows for users to initialize all their state variables and perform other init actions before handling data.
Function that will be invoked as the first method that allows for users to initialize all their state variables and perform other init actions before handling data.
- outputMode
\- output mode for the stateful processor
- timeMode
\- time mode for the stateful processor.
- Definition Classes
- StatefulProcessor
Concrete Value Members
- object implicits extends EncoderImplicits
- Definition Classes
- StatefulProcessor
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def close(): Unit
Function called as the last method that allows for users to perform any cleanup or teardown operations.
Function called as the last method that allows for users to perform any cleanup or teardown operations.
- Definition Classes
- StatefulProcessor
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def getHandle: StatefulProcessorHandle
Function to get the stateful processor handle that will be used to interact with the state
Function to get the stateful processor handle that will be used to interact with the state
- returns
handle - instance of StatefulProcessorHandle
- Definition Classes
- StatefulProcessor
- def handleExpiredTimer(key: K, timerValues: TimerValues, expiredTimerInfo: ExpiredTimerInfo): Iterator[O]
Function that will be invoked when a timer is fired for a given key.
Function that will be invoked when a timer is fired for a given key. Users can choose to evict state, register new timers and optionally provide output rows.
Note that in microbatch mode, this function will be called once for each unique timer expiry for a given key. If no timer expires for a given key, this function will not be invoked for that key.
- key
\- grouping key
- timerValues
\- instance of TimerValues that provides access to current processing/event
- expiredTimerInfo
\- instance of ExpiredTimerInfo that provides access to expired timer
- returns
Zero or more output rows
- Definition Classes
- StatefulProcessor
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def setHandle(handle: StatefulProcessorHandle): Unit
Function to set the stateful processor handle that will be used to interact with the state store and other stateful processor related operations.
Function to set the stateful processor handle that will be used to interact with the state store and other stateful processor related operations.
- handle
\- instance of StatefulProcessorHandle
- Definition Classes
- StatefulProcessor
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)