org.apache.arrow.vector.complex

Class BaseRepeatedValueVector

java.lang.Object

org.apache.arrow.vector.BaseValueVector

org.apache.arrow.vector.complex.BaseRepeatedValueVector

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<ValueVector>, BaseListVector, RepeatedValueVector, DensityAwareVector, FieldVector, ValueVector

Direct Known Subclasses:

ListVector

public abstract class BaseRepeatedValueVector
extends BaseValueVector
implements RepeatedValueVector, BaseListVector

Base class for Vectors that contain repeated values.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATA_VECTOR_NAME
static FieldVector	DEFAULT_DATA_VECTOR
protected String	defaultDataVectorName
static byte	OFFSET_WIDTH
protected long	offsetAllocationSizeInBytes
protected ArrowBuf	offsetBuffer
protected CallBack	repeatedCallBack
protected int	valueCount
protected FieldVector	vector

Fields inherited from class org.apache.arrow.vector.BaseValueVector

allocator, fieldReader, INITIAL_VALUE_ALLOCATION, MAX_ALLOCATION_SIZE, MAX_ALLOCATION_SIZE_PROPERTY

Fields inherited from interface org.apache.arrow.vector.complex.RepeatedValueVector

DEFAULT_REPEAT_PER_RECORD

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseRepeatedValueVector(String name, BufferAllocator allocator, CallBack callBack)
protected	BaseRepeatedValueVector(String name, BufferAllocator allocator, FieldVector vector, CallBack callBack)

Method Summary

Modifier and Type	Method and Description
<T extends ValueVector> AddOrGetResult<T>	addOrGetVector(FieldType fieldType) Initialize the data vector (and execute callback) if it hasn't already been done, returns the data vector.
boolean	allocateNewSafe() Allocates new buffers.
protected ArrowBuf	allocateOffsetBuffer(long size)
void	clear() Release any owned ArrowBuf and reset the ValueVector to the initial state.
ArrowBuf[]	getBuffers(boolean clear) Return the underlying buffers associated with this vector.
int	getBufferSize() Get the number of bytes used by this vector.
int	getBufferSizeFor(int valueCount) Returns the number of bytes that is used by this vector if it holds the given number of values.
FieldVector	getDataVector() Get the data vector.
int	getInnerValueCount()
int	getInnerValueCountAt(int index) Returns the value count for inner data vector at a particular index.
String	getName() Gets the name of the vector.
protected int	getOffsetBufferValueCapacity()
UInt4Vector	getOffsetVector() Deprecated. This API will be removed, as the current implementations no longer hold inner offset vectors.
int	getValueCapacity() Returns the maximum number of values that can be stored in this vector instance.
int	getValueCount() Gets the number of values.
boolean	isEmpty(int index) Return if value at index is empty (this implementation is always false).
boolean	isNull(int index) Return if value at index is null (this implementation is always false).
Iterator<ValueVector>	iterator()
void	reAlloc() Allocate new buffer with double capacity, and copy data into the new buffer.
protected void	reallocOffsetBuffer()
protected void	replaceDataVector(FieldVector v)
void	reset() Reset the ValueVector to the initial state without releasing any owned ArrowBuf.
void	setInitialCapacity(int numRecords) Set the initial record capacity.
void	setInitialCapacity(int numRecords, double density) Specialized version of setInitialCapacity() for ListVector.
void	setInitialTotalCapacity(int numRecords, int totalNumberOfElements) Specialized version of setInitialTotalCapacity() for ListVector.
void	setValueCount(int valueCount) Preallocates the number of repeated values.
int	size() Get value indicating if inner vector is set.
int	startNewValue(int index) Starts a new repeated value.

Methods inherited from class org.apache.arrow.vector.BaseValueVector

checkBufRefs, close, copyFrom, copyFromSafe, getAllocator, getReader, getReaderImpl, getTransferPair, getValidityBufferSizeFromCount, releaseBuffer, toString, transferBuffer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.arrow.vector.complex.BaseListVector

getElementEndIndex, getElementStartIndex

Methods inherited from interface org.apache.arrow.vector.FieldVector

exportBuffer, exportCDataBuffers, getChildrenFromFields, getDataBufferAddress, getFieldBuffers, getFieldInnerVectors, getOffsetBufferAddress, getValidityBufferAddress, initializeChildrenFromFields, loadFieldBuffers, setNull

Methods inherited from interface org.apache.arrow.vector.ValueVector

accept, allocateNew, close, copyFrom, copyFromSafe, getAllocator, getDataBuffer, getField, getMinorType, getNullCount, getObject, getOffsetBuffer, getReader, getTransferPair, getTransferPair, getTransferPair, getTransferPair, getTransferPair, getValidityBuffer, hashCode, hashCode, makeTransferPair, validate, validateFull

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

DEFAULT_DATA_VECTOR

public static final FieldVector DEFAULT_DATA_VECTOR

DATA_VECTOR_NAME

public static final String DATA_VECTOR_NAME

See Also:

Constant Field Values

OFFSET_WIDTH

public static final byte OFFSET_WIDTH

See Also:

Constant Field Values

offsetBuffer

protected ArrowBuf offsetBuffer

vector

protected FieldVector vector

repeatedCallBack

protected final CallBack repeatedCallBack

valueCount

protected int valueCount

offsetAllocationSizeInBytes

protected long offsetAllocationSizeInBytes

defaultDataVectorName

protected String defaultDataVectorName

Constructor Detail

BaseRepeatedValueVector

protected BaseRepeatedValueVector(String name,
                                  BufferAllocator allocator,
                                  CallBack callBack)

BaseRepeatedValueVector

protected BaseRepeatedValueVector(String name,
                                  BufferAllocator allocator,
                                  FieldVector vector,
                                  CallBack callBack)

Method Detail

getName

public String getName()

Description copied from interface: ValueVector

Gets the name of the vector.

Specified by:

getName in interface ValueVector

Specified by:

getName in class BaseValueVector

Returns:

the name of the vector.

allocateNewSafe

public boolean allocateNewSafe()

Description copied from interface: ValueVector

Allocates new buffers. ValueVector implements logic to determine how much to allocate.

Specified by:

allocateNewSafe in interface ValueVector

Returns:

Returns true if allocation was successful.

allocateOffsetBuffer

protected ArrowBuf allocateOffsetBuffer(long size)

reAlloc

public void reAlloc()

Description copied from interface: ValueVector

Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's buffer with new buffer, and release old one

Specified by:

reAlloc in interface ValueVector

reallocOffsetBuffer

protected void reallocOffsetBuffer()

getOffsetVector

@Deprecated
public UInt4Vector getOffsetVector()

Deprecated. This API will be removed, as the current implementations no longer hold inner offset vectors.

Get the offset vector.

Specified by:

getOffsetVector in interface RepeatedValueVector

Returns:

the underlying offset vector or null if none exists.

getDataVector

public FieldVector getDataVector()

Description copied from interface: RepeatedValueVector

Get the data vector.

Specified by:

getDataVector in interface RepeatedValueVector

Returns:

the underlying data vector or null if none exists.

setInitialCapacity

public void setInitialCapacity(int numRecords)

Description copied from interface: ValueVector

Set the initial record capacity.

Specified by:

setInitialCapacity in interface ValueVector

Parameters:

numRecords - the initial record capacity.

setInitialCapacity

public void setInitialCapacity(int numRecords,
                               double density)

Specialized version of setInitialCapacity() for ListVector. This is used by some callers when they want to explicitly control and be conservative about memory allocated for inner data vector. This is very useful when we are working with memory constraints for a query and have a fixed amount of memory reserved for the record batch. In such cases, we are likely to face OOM or related problems when we reserve memory for a record batch with value count x and do setInitialCapacity(x) such that each vector allocates only what is necessary and not the default amount but the multiplier forces the memory requirement to go beyond what was needed.

Specified by:

setInitialCapacity in interface DensityAwareVector

Parameters:

numRecords - value count

density - density of ListVector. Density is the average size of list per position in the List vector. For example, a density value of 10 implies each position in the list vector has a list of 10 values. A density value of 0.1 implies out of 10 positions in the list vector, 1 position has a list of size 1 and remaining positions are null (no lists) or empty lists. This helps in tightly controlling the memory we provision for inner data vector.

setInitialTotalCapacity

public void setInitialTotalCapacity(int numRecords,
                                    int totalNumberOfElements)

Specialized version of setInitialTotalCapacity() for ListVector. This is used by some callers when they want to explicitly control and be conservative about memory allocated for inner data vector. This is very useful when we are working with memory constraints for a query and have a fixed amount of memory reserved for the record batch. In such cases, we are likely to face OOM or related problems when we reserve memory for a record batch with value count x and do setInitialCapacity(x) such that each vector allocates only what is necessary and not the default amount but the multiplier forces the memory requirement to go beyond what was needed.

Parameters:

numRecords - value count

totalNumberOfElements - the total number of elements to to allow for in this vector across all records.

getValueCapacity

public int getValueCapacity()

Description copied from interface: ValueVector

Returns the maximum number of values that can be stored in this vector instance.

Specified by:

getValueCapacity in interface ValueVector

Returns:

the maximum number of values that can be stored in this vector instance.

getOffsetBufferValueCapacity

protected int getOffsetBufferValueCapacity()

getBufferSize

public int getBufferSize()

Description copied from interface: ValueVector

Get the number of bytes used by this vector.

Specified by:

getBufferSize in interface ValueVector

Returns:

the number of bytes that is used by this vector instance.

getBufferSizeFor

public int getBufferSizeFor(int valueCount)

Description copied from interface: ValueVector

Returns the number of bytes that is used by this vector if it holds the given number of values. The result will be the same as if setValueCount() were called, followed by calling getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt finishing off the population of a vector. Some operations might wish to use this to determine how much memory has been used by a vector so far, even though it is not finished being populated.

Specified by:

getBufferSizeFor in interface ValueVector

Parameters:

valueCount - the number of values to assume this vector contains

Returns:

the buffer size if this vector is holding valueCount values

iterator

public Iterator<ValueVector> iterator()

Specified by:

iterator in interface Iterable<ValueVector>

Overrides:

iterator in class BaseValueVector

clear

public void clear()

Description copied from interface: ValueVector

Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has any child vectors, they will also be cleared.

Specified by:

clear in interface ValueVector

Overrides:

clear in class BaseValueVector

reset

public void reset()

Description copied from interface: ValueVector

Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer capacities will remain unchanged and any previous data will be zeroed out. This includes buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be reset.

Specified by:

reset in interface ValueVector

getBuffers

public ArrowBuf[] getBuffers(boolean clear)

Description copied from interface: ValueVector

Return the underlying buffers associated with this vector. Note that this doesn't impact the reference counts for this buffer so it only should be used for in-context access. Also note that this buffer changes regularly thus external classes shouldn't hold a reference to it (unless they change it).

Specified by:

getBuffers in interface ValueVector

Parameters:

clear - Whether to clear vector before returning; the buffers will still be refcounted; but the returned array will be the only reference to them

Returns:

The underlying buffers that is used by this vector instance.

size

public int size()

Get value indicating if inner vector is set.

Returns:

1 if inner vector is explicitly set via #addOrGetVector else 0

addOrGetVector

public <T extends ValueVector> AddOrGetResult<T> addOrGetVector(FieldType fieldType)

Initialize the data vector (and execute callback) if it hasn't already been done, returns the data vector.

replaceDataVector

protected void replaceDataVector(FieldVector v)

getValueCount

public int getValueCount()

Description copied from interface: ValueVector

Gets the number of values.

Specified by:

getValueCount in interface ValueVector

Returns:

number of values in the vector

getInnerValueCount

public int getInnerValueCount()

getInnerValueCountAt

public int getInnerValueCountAt(int index)

Returns the value count for inner data vector at a particular index.

isNull

public boolean isNull(int index)

Return if value at index is null (this implementation is always false).

Specified by:

isNull in interface ValueVector

Parameters:

index - index to check for null

Returns:

true if element is null

isEmpty

public boolean isEmpty(int index)

Return if value at index is empty (this implementation is always false).

startNewValue

public int startNewValue(int index)

Starts a new repeated value.

setValueCount

public void setValueCount(int valueCount)

Preallocates the number of repeated values.

Specified by:

setValueCount in interface ValueVector